[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:27:19 dist-test-slave-dfzm krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:27:19 dist-test-slave-dfzm krb5kdc[399](info): set up 2 sockets
Sep 26 09:27:19 dist-test-slave-dfzm krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 09:27:20.125818 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:27:20 dist-test-slave-dfzm krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:27:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:27:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:27:20.245465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32947
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:20.324744 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:20.324996 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:20.325032 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:20.326188 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:27:20.326268 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:20.326294 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:20.326318 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:27:20.326345 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:27:20.327816 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:20.328099 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:20.328347 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:20.333433 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:20.333494 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:20.333496 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:20.333770 412 server_base.cc:1034] running on GCE node
I20260926 09:27:20.334152 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:20.334419 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:20.335592 412 hybrid_clock.cc:648] HybridClock initialized: now 1790414840335580 us; error 35 us; skew 500 ppm
Sep 26 09:27:20 dist-test-slave-dfzm krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:20.338258 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:27:20.338752 412 webserver.cc:460] Webserver started at http://127.0.98.62:38245/ using document root <none> and password file <none>
I20260926 09:27:20.338987 412 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:20.339035 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:20.339125 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:20.340186 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "7eeb1c30f1ea4e3586e7c4da051c409d"
format_stamp: "Formatted at 2026-09-26 09:27:20 on dist-test-slave-dfzm"
server_key: "84cc02c36de2d743c04095b2c945d01f"
server_key_iv: "187028488d54a5edc979d50931bdfb76"
server_key_version: "encryptionkey@0"
I20260926 09:27:20.340493 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "7eeb1c30f1ea4e3586e7c4da051c409d"
format_stamp: "Formatted at 2026-09-26 09:27:20 on dist-test-slave-dfzm"
server_key: "84cc02c36de2d743c04095b2c945d01f"
server_key_iv: "187028488d54a5edc979d50931bdfb76"
server_key_version: "encryptionkey@0"
I20260926 09:27:20.342010 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:27:20.342749 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:20.342975 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:20.343093 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "7eeb1c30f1ea4e3586e7c4da051c409d"
format_stamp: "Formatted at 2026-09-26 09:27:20 on dist-test-slave-dfzm"
server_key: "84cc02c36de2d743c04095b2c945d01f"
server_key_iv: "187028488d54a5edc979d50931bdfb76"
server_key_version: "encryptionkey@0"
I20260926 09:27:20.343179 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:20.356639 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:20.358749 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:20.358965 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:20.363099 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32947
I20260926 09:27:20.363134 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32947 every 8 connection(s)
I20260926 09:27:20.363510 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:27:20.364378 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:20.366675 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eeb1c30f1ea4e3586e7c4da051c409d: Bootstrap starting.
I20260926 09:27:20.367192 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eeb1c30f1ea4e3586e7c4da051c409d: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:20.367424 480 log.cc:824] T 00000000000000000000000000000000 P 7eeb1c30f1ea4e3586e7c4da051c409d: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:20.368072 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eeb1c30f1ea4e3586e7c4da051c409d: No bootstrap required, opened a new log
I20260926 09:27:20.369642 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eeb1c30f1ea4e3586e7c4da051c409d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeb1c30f1ea4e3586e7c4da051c409d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32947 } }
I20260926 09:27:20.369796 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7eeb1c30f1ea4e3586e7c4da051c409d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:20.369819 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eeb1c30f1ea4e3586e7c4da051c409d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eeb1c30f1ea4e3586e7c4da051c409d, State: Initialized, Role: FOLLOWER
I20260926 09:27:20.369992 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eeb1c30f1ea4e3586e7c4da051c409d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeb1c30f1ea4e3586e7c4da051c409d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32947 } }
I20260926 09:27:20.369979 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 412
I20260926 09:27:20.370065 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eeb1c30f1ea4e3586e7c4da051c409d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:27:20.370097 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:27:20.370115 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eeb1c30f1ea4e3586e7c4da051c409d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:27:20.370164 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eeb1c30f1ea4e3586e7c4da051c409d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:20.370711 392 external_mini_cluster.cc:1428] Setting key aee628e947c8fd69ea6abf98e36ffa35
I20260926 09:27:20.370707 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eeb1c30f1ea4e3586e7c4da051c409d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeb1c30f1ea4e3586e7c4da051c409d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32947 } }
I20260926 09:27:20.370781 480 leader_election.cc:302] T 00000000000000000000000000000000 P 7eeb1c30f1ea4e3586e7c4da051c409d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eeb1c30f1ea4e3586e7c4da051c409d; no voters:
I20260926 09:27:20.370962 480 leader_election.cc:288] T 00000000000000000000000000000000 P 7eeb1c30f1ea4e3586e7c4da051c409d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:27:20.371033 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eeb1c30f1ea4e3586e7c4da051c409d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:20.371246 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eeb1c30f1ea4e3586e7c4da051c409d [term 1 LEADER]: Becoming Leader. State: Replica: 7eeb1c30f1ea4e3586e7c4da051c409d, State: Running, Role: LEADER
I20260926 09:27:20.371308 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eeb1c30f1ea4e3586e7c4da051c409d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:27:20.371402 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eeb1c30f1ea4e3586e7c4da051c409d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeb1c30f1ea4e3586e7c4da051c409d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32947 } }
I20260926 09:27:20.372045 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eeb1c30f1ea4e3586e7c4da051c409d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7eeb1c30f1ea4e3586e7c4da051c409d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeb1c30f1ea4e3586e7c4da051c409d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32947 } } }
I20260926 09:27:20.372139 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eeb1c30f1ea4e3586e7c4da051c409d [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:20.372323 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eeb1c30f1ea4e3586e7c4da051c409d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eeb1c30f1ea4e3586e7c4da051c409d. Latest consensus state: current_term: 1 leader_uuid: "7eeb1c30f1ea4e3586e7c4da051c409d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeb1c30f1ea4e3586e7c4da051c409d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32947 } } }
I20260926 09:27:20.372421 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eeb1c30f1ea4e3586e7c4da051c409d [sys.catalog]: This master's current role is: LEADER
W20260926 09:27:20.373236 501 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7eeb1c30f1ea4e3586e7c4da051c409d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:27:20.373344 501 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:27:20.373469 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:27:20.373677 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:27:20.375223 494 catalog_manager.cc:1347] Generated new cluster ID: 8efafb47dae84ee9b16306670bf1f81c
I20260926 09:27:20.375278 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:27:20 dist-test-slave-dfzm krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:20.392973 488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:20.371681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59802 (local address 127.0.98.62:32947)
0926 09:27:20.372548 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:20.372559 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:20.372598 (+ 39us) server_negotiation.cc:407] Connection header received
0926 09:27:20.374498 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:20.374505 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:20.374571 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:20.374713 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:20.381106 (+ 6393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:20.382624 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:20.384372 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:20.384996 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:20.385091 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:20.389251 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:20.389256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:20.389256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:20.389269 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:20.391072 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:20.391531 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:20.391534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:20.391535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:20.391633 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:20.391890 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:20.391891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:20.391891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:20.392150 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:20.392319 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:20.392599 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 09:27:20.392744 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":84,"threads_started":1}
I20260926 09:27:20.405714 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:27:20.406296 494 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:27:20.414736 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7eeb1c30f1ea4e3586e7c4da051c409d: Generated new TSK 0
I20260926 09:27:20.414968 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:27:20.450672 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32947
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:20.549335 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:20.549538 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:20.549562 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:20.550753 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:20.550801 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:20.550843 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:27:20.552253 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32947
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:20.552454 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:20.552634 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:20.557526 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:20.557523 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:20.557519 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:20.557888 506 server_base.cc:1034] running on GCE node
I20260926 09:27:20.558095 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:20.558334 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:20.559509 506 hybrid_clock.cc:648] HybridClock initialized: now 1790414840559492 us; error 28 us; skew 500 ppm
Sep 26 09:27:20 dist-test-slave-dfzm krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:20.562189 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:27:20.562677 506 webserver.cc:460] Webserver started at http://127.0.98.1:34045/ using document root <none> and password file <none>
I20260926 09:27:20.562923 506 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:20.562996 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:20.563149 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:20.564206 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "ec8a5f422d4e4900a21f8112b1c1e65b"
format_stamp: "Formatted at 2026-09-26 09:27:20 on dist-test-slave-dfzm"
server_key: "303007bd8b61da0b1f4a2e48d94add56"
server_key_iv: "12f7ed095a20e405befa310e649a3ee2"
server_key_version: "encryptionkey@0"
I20260926 09:27:20.564610 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ec8a5f422d4e4900a21f8112b1c1e65b"
format_stamp: "Formatted at 2026-09-26 09:27:20 on dist-test-slave-dfzm"
server_key: "303007bd8b61da0b1f4a2e48d94add56"
server_key_iv: "12f7ed095a20e405befa310e649a3ee2"
server_key_version: "encryptionkey@0"
I20260926 09:27:20.566010 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:27:20.566771 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:20.566958 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:20.567057 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "ec8a5f422d4e4900a21f8112b1c1e65b"
format_stamp: "Formatted at 2026-09-26 09:27:20 on dist-test-slave-dfzm"
server_key: "303007bd8b61da0b1f4a2e48d94add56"
server_key_iv: "12f7ed095a20e405befa310e649a3ee2"
server_key_version: "encryptionkey@0"
I20260926 09:27:20.567113 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:20.582679 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:20.585062 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:20.585175 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:20.585366 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:20.585767 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:20.585827 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:20.585868 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:20.585894 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:20.591799 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42799
I20260926 09:27:20.591876 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42799 every 8 connection(s)
I20260926 09:27:20.592239 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:27:20.596207 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 506
I20260926 09:27:20.596298 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:27:20.596436 392 external_mini_cluster.cc:1428] Setting key 1a1a2d97a14bf02135600462f360f77c
Sep 26 09:27:20 dist-test-slave-dfzm krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:20.603022 488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:20.593060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51713 (local address 127.0.98.62:32947)
0926 09:27:20.593130 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:20.593132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:20.593344 (+ 212us) server_negotiation.cc:407] Connection header received
0926 09:27:20.593899 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:20.593900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:20.593909 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:20.593970 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:20.594915 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:20.595403 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:20.596670 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:20.596904 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:20.596955 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:20.599707 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:20.599709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:20.599709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:20.599717 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:20.601358 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:20.601886 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:20.601887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:20.601887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:20.601955 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:20.602255 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:20.602256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:20.602256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:20.602425 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:20.602504 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:20.602778 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 09:27:20.602878 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 09:27:20.603044 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:20.593040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32947 (local address 127.0.98.1:51713)
0926 09:27:20.593210 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:20.593254 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:20.593805 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:20.594028 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:20.594032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:20.594628 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:20.594824 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:20.594829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:20.595491 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:20.595492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:20.596561 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:20.596565 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:20.596977 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:20.596978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:20.597120 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:20.597754 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:20.597765 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:20.599613 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:20.601439 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:20.601450 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:20.601452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:20.601779 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:20.602048 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:20.602049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:20.602049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:20.602157 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:20.602500 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:20.602505 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:20.602627 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 09:27:20.602710 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 09:27:20.602878 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260926 09:27:20.603574 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32947
I20260926 09:27:20.603693 635 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:20.603900 635 heartbeater.cc:511] Master 127.0.98.62:32947 requested a full tablet report, sending...
I20260926 09:27:20.604532 444 ts_manager.cc:194] Registered new tserver with Master: ec8a5f422d4e4900a21f8112b1c1e65b (127.0.98.1:42799)
I20260926 09:27:20.605508 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51713
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:27:20.623942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32947
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:20.704849 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:20.705060 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:20.705083 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:20.706305 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:20.706347 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:20.706401 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:27:20.707847 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32947
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:20.708091 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:20.708325 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:20.713233 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:20.713313 640 server_base.cc:1034] running on GCE node
W20260926 09:27:20.713289 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:20.713231 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:20.713670 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:20.713935 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:20.715121 640 hybrid_clock.cc:648] HybridClock initialized: now 1790414840715116 us; error 39 us; skew 500 ppm
Sep 26 09:27:20 dist-test-slave-dfzm krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:20.717715 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:27:20.718218 640 webserver.cc:460] Webserver started at http://127.0.98.2:43851/ using document root <none> and password file <none>
I20260926 09:27:20.718461 640 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:20.718531 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:20.718670 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:20.719812 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "3515949196584e3bb16ed45eab1c46c8"
format_stamp: "Formatted at 2026-09-26 09:27:20 on dist-test-slave-dfzm"
server_key: "1b805ae8bbd78b377c9b12c434a1d14d"
server_key_iv: "abdcce4d886a89d8f4c4330247be8825"
server_key_version: "encryptionkey@0"
I20260926 09:27:20.720206 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3515949196584e3bb16ed45eab1c46c8"
format_stamp: "Formatted at 2026-09-26 09:27:20 on dist-test-slave-dfzm"
server_key: "1b805ae8bbd78b377c9b12c434a1d14d"
server_key_iv: "abdcce4d886a89d8f4c4330247be8825"
server_key_version: "encryptionkey@0"
I20260926 09:27:20.721658 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260926 09:27:20.722481 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:20.722684 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:20.722791 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "3515949196584e3bb16ed45eab1c46c8"
format_stamp: "Formatted at 2026-09-26 09:27:20 on dist-test-slave-dfzm"
server_key: "1b805ae8bbd78b377c9b12c434a1d14d"
server_key_iv: "abdcce4d886a89d8f4c4330247be8825"
server_key_version: "encryptionkey@0"
I20260926 09:27:20.722880 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:20.739627 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:20.741662 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:20.741827 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:20.742082 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:20.742398 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:20.742452 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:20.742487 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:20.742534 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:20.749634 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37445
I20260926 09:27:20.749661 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37445 every 8 connection(s)
I20260926 09:27:20.750011 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:27:20 dist-test-slave-dfzm krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:20.758908 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 640
I20260926 09:27:20.759011 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:27:20.759184 392 external_mini_cluster.cc:1428] Setting key 31aa70c291fda11d56b138ee1e8bfb67
I20260926 09:27:20.759407 488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:20.750812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34821 (local address 127.0.98.62:32947)
0926 09:27:20.750899 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:20.750901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:20.751101 (+ 200us) server_negotiation.cc:407] Connection header received
0926 09:27:20.751542 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:20.751544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:20.751557 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:20.751613 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:20.752561 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:20.753041 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:20.754123 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:20.754310 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:20.754366 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:20.756522 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:20.756524 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:20.756524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:20.756531 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:20.757921 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:20.758325 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:20.758326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:20.758327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:20.758384 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:20.758611 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:20.758611 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:20.758612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:20.758715 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:20.758765 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:20.759180 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 09:27:20.759283 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 09:27:20.759567 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:20.750768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32947 (local address 127.0.98.2:34821)
0926 09:27:20.750954 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:20.750993 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:20.751457 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:20.751683 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:20.751686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:20.752260 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:20.752474 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:20.752479 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:20.753129 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:20.753130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:20.754044 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:20.754046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:20.754386 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:20.754387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:20.754499 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:20.754977 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:20.754985 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:20.756431 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:20.758018 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:20.758025 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:20.758026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:20.758251 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:20.758446 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:20.758447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:20.758447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:20.758554 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:20.758796 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:20.758804 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:20.758990 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 09:27:20.759155 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 09:27:20.759334 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":98,"threads_started":1}
I20260926 09:27:20.760035 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32947
I20260926 09:27:20.760119 769 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:20.760310 769 heartbeater.cc:511] Master 127.0.98.62:32947 requested a full tablet report, sending...
I20260926 09:27:20.760684 444 ts_manager.cc:194] Registered new tserver with Master: 3515949196584e3bb16ed45eab1c46c8 (127.0.98.2:37445)
I20260926 09:27:20.761193 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34821
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:27:20.784972 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32947
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:20.867589 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:20.867815 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:20.867856 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:20.869050 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:20.869122 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:20.869174 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:27:20.870921 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32947
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:20.871183 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:20.871426 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:27:20.876734 774 server_base.cc:1034] running on GCE node
W20260926 09:27:20.876709 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:20.876714 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:20.876783 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:20.877097 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:20.877359 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:20.878526 774 hybrid_clock.cc:648] HybridClock initialized: now 1790414840878505 us; error 44 us; skew 500 ppm
Sep 26 09:27:20 dist-test-slave-dfzm krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:20.881244 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:27:20.881752 774 webserver.cc:460] Webserver started at http://127.0.98.3:43897/ using document root <none> and password file <none>
I20260926 09:27:20.882067 774 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:20.882124 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:20.882216 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:20.883247 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "b8456f46fc9d4fdc89fc88f21c1d5241"
format_stamp: "Formatted at 2026-09-26 09:27:20 on dist-test-slave-dfzm"
server_key: "287f362bc652b308e6e08c7cb49b488f"
server_key_iv: "d99c189addf7dcfbee12a7ffc94de934"
server_key_version: "encryptionkey@0"
I20260926 09:27:20.883584 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b8456f46fc9d4fdc89fc88f21c1d5241"
format_stamp: "Formatted at 2026-09-26 09:27:20 on dist-test-slave-dfzm"
server_key: "287f362bc652b308e6e08c7cb49b488f"
server_key_iv: "d99c189addf7dcfbee12a7ffc94de934"
server_key_version: "encryptionkey@0"
I20260926 09:27:20.884964 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:27:20.885782 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:20.885993 774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:20.886106 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "b8456f46fc9d4fdc89fc88f21c1d5241"
format_stamp: "Formatted at 2026-09-26 09:27:20 on dist-test-slave-dfzm"
server_key: "287f362bc652b308e6e08c7cb49b488f"
server_key_iv: "d99c189addf7dcfbee12a7ffc94de934"
server_key_version: "encryptionkey@0"
I20260926 09:27:20.886193 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:20.910580 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:20.912928 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:20.913125 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:20.913367 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:20.913746 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:20.913784 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:20.913807 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:20.913841 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:20.920536 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46753
I20260926 09:27:20.920583 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46753 every 8 connection(s)
I20260926 09:27:20.920923 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:27:20 dist-test-slave-dfzm krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:20.929941 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 774
I20260926 09:27:20.930054 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:27:20.930212 392 external_mini_cluster.cc:1428] Setting key 02551c01ec789922cccaa6569eb162a5
I20260926 09:27:20.930217 488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:20.921712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37345 (local address 127.0.98.62:32947)
0926 09:27:20.921767 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:20.921769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:20.921994 (+ 225us) server_negotiation.cc:407] Connection header received
0926 09:27:20.922431 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:20.922433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:20.922443 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:20.922499 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:20.923411 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:20.923924 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:20.924883 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:20.925091 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:20.925149 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:20.927433 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:20.927434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:20.927434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:20.927441 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:20.928652 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:20.929016 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:20.929017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:20.929017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:20.929066 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:20.929306 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:20.929307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:20.929307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:20.929444 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:20.929509 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:20.930038 (+ 529us) server_negotiation.cc:299] Negotiation successful
0926 09:27:20.930125 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 09:27:20.930505 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:20.921690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32947 (local address 127.0.98.3:37345)
0926 09:27:20.921841 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:20.921878 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:20.922336 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:20.922552 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:20.922555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:20.923114 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:20.923304 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:20.923309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:20.924008 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:20.924009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:20.924803 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:20.924804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:20.925167 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:20.925168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:20.925275 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:20.925863 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:20.925878 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:20.927345 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:20.928725 (+ 1380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:20.928730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:20.928732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:20.928949 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:20.929125 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:20.929126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:20.929126 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:20.929224 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:20.929532 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:20.929538 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:20.929737 (+ 199us) client_negotiation.cc:769] Sending connection context
0926 09:27:20.929993 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 09:27:20.930270 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260926 09:27:20.931125 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32947
I20260926 09:27:20.931252 903 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:20.931435 903 heartbeater.cc:511] Master 127.0.98.62:32947 requested a full tablet report, sending...
I20260926 09:27:20.931818 444 ts_manager.cc:194] Registered new tserver with Master: b8456f46fc9d4fdc89fc88f21c1d5241 (127.0.98.3:46753)
I20260926 09:27:20.932186 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37345
I20260926 09:27:20.942513 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 09:27:20 dist-test-slave-dfzm krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414840, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 09:27:20 dist-test-slave-dfzm krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414840, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 09:27:20.966436 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:20.958117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54868 (local address 127.0.98.1:42799)
0926 09:27:20.958331 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:20.958336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:20.958348 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:27:20.958378 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:20.958380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:20.958395 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:20.958471 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:20.959009 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:20.959801 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:20.960598 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:20.960962 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:20.961028 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:20.963342 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:20.963350 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:20.963350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:20.963357 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:20.965237 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:20.965689 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:20.965693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:20.965693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:20.965753 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:20.965916 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:20.965917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:20.965917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:20.966040 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:20.966169 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:20.966238 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 09:27:20.966303 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":107,"threads_started":1}
W20260926 09:27:20.966876 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54868
Sep 26 09:27:20 dist-test-slave-dfzm krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 09:27:20 dist-test-slave-dfzm krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 09:27:20.989905 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:20.982747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54882 (local address 127.0.98.1:42799)
0926 09:27:20.982806 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:20.982812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:20.982822 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:27:20.982878 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:20.982879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:20.982889 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:20.982938 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:20.983538 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:20.984032 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:20.984829 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:20.985035 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:20.985085 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:20.987328 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:20.987329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:20.987330 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:20.987336 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:20.988819 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:20.989111 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:20.989112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:20.989112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:20.989162 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:20.989352 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:20.989354 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:20.989354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:20.989480 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:20.989630 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:20.989772 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 09:27:20.989839 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 09:27:20.990439 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 506
I20260926 09:27:20.995950 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 640
I20260926 09:27:21.001449 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 774
I20260926 09:27:21.006223 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 412
2026-09-26T09:27:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1943 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:27:21 dist-test-slave-dfzm krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:27:21 dist-test-slave-dfzm krb5kdc[917](info): set up 2 sockets
Sep 26 09:27:21 dist-test-slave-dfzm krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260926 09:27:22.041157 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:27:22 dist-test-slave-dfzm krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:27:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:27:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:27:22.160740 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37509
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:22.242116 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:22.242390 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:22.242436 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:22.243615 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:27:22.243693 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:22.243723 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:22.243760 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:27:22.243788 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:27:22.245301 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:22.245626 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:22.245888 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:22.250751 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:22.250782 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:22.250864 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:22.251343 930 server_base.cc:1034] running on GCE node
I20260926 09:27:22.251634 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:22.251917 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:22.253082 930 hybrid_clock.cc:648] HybridClock initialized: now 1790414842253063 us; error 37 us; skew 500 ppm
Sep 26 09:27:22 dist-test-slave-dfzm krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:22.255861 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:27:22.256373 930 webserver.cc:460] Webserver started at http://127.0.98.62:44849/ using document root <none> and password file <none>
I20260926 09:27:22.256650 930 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:22.256728 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:22.256888 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:22.258112 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "e8c68904d6a84ad3a9f4a6a6906d1a78"
format_stamp: "Formatted at 2026-09-26 09:27:22 on dist-test-slave-dfzm"
server_key: "d240834c8ddded7d85597c09b8ca0660"
server_key_iv: "df3120e8d4aa9f84397750eb0a49b767"
server_key_version: "encryptionkey@0"
I20260926 09:27:22.258498 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "e8c68904d6a84ad3a9f4a6a6906d1a78"
format_stamp: "Formatted at 2026-09-26 09:27:22 on dist-test-slave-dfzm"
server_key: "d240834c8ddded7d85597c09b8ca0660"
server_key_iv: "df3120e8d4aa9f84397750eb0a49b767"
server_key_version: "encryptionkey@0"
I20260926 09:27:22.259929 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:27:22.260717 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:22.261003 930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:22.261118 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "e8c68904d6a84ad3a9f4a6a6906d1a78"
format_stamp: "Formatted at 2026-09-26 09:27:22 on dist-test-slave-dfzm"
server_key: "d240834c8ddded7d85597c09b8ca0660"
server_key_iv: "df3120e8d4aa9f84397750eb0a49b767"
server_key_version: "encryptionkey@0"
I20260926 09:27:22.261200 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:22.294067 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:22.296219 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:22.296337 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:22.300439 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37509
I20260926 09:27:22.300506 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37509 every 8 connection(s)
I20260926 09:27:22.300874 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:27:22.301848 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:22.304064 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8c68904d6a84ad3a9f4a6a6906d1a78: Bootstrap starting.
I20260926 09:27:22.304632 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8c68904d6a84ad3a9f4a6a6906d1a78: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:22.304927 998 log.cc:824] T 00000000000000000000000000000000 P e8c68904d6a84ad3a9f4a6a6906d1a78: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:22.305701 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8c68904d6a84ad3a9f4a6a6906d1a78: No bootstrap required, opened a new log
I20260926 09:27:22.305701 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 930
I20260926 09:27:22.305800 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:27:22.305958 392 external_mini_cluster.cc:1428] Setting key f86aa966a7f7c757af73562392e02c4a
I20260926 09:27:22.308320 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8c68904d6a84ad3a9f4a6a6906d1a78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c68904d6a84ad3a9f4a6a6906d1a78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37509 } }
I20260926 09:27:22.308477 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8c68904d6a84ad3a9f4a6a6906d1a78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:22.308524 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8c68904d6a84ad3a9f4a6a6906d1a78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8c68904d6a84ad3a9f4a6a6906d1a78, State: Initialized, Role: FOLLOWER
I20260926 09:27:22.308663 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8c68904d6a84ad3a9f4a6a6906d1a78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c68904d6a84ad3a9f4a6a6906d1a78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37509 } }
I20260926 09:27:22.308759 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8c68904d6a84ad3a9f4a6a6906d1a78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:27:22.308804 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8c68904d6a84ad3a9f4a6a6906d1a78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:27:22.308851 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8c68904d6a84ad3a9f4a6a6906d1a78 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:22.309392 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8c68904d6a84ad3a9f4a6a6906d1a78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c68904d6a84ad3a9f4a6a6906d1a78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37509 } }
I20260926 09:27:22.309507 998 leader_election.cc:302] T 00000000000000000000000000000000 P e8c68904d6a84ad3a9f4a6a6906d1a78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8c68904d6a84ad3a9f4a6a6906d1a78; no voters:
I20260926 09:27:22.309713 998 leader_election.cc:288] T 00000000000000000000000000000000 P e8c68904d6a84ad3a9f4a6a6906d1a78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:27:22.309785 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8c68904d6a84ad3a9f4a6a6906d1a78 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:22.309934 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8c68904d6a84ad3a9f4a6a6906d1a78 [term 1 LEADER]: Becoming Leader. State: Replica: e8c68904d6a84ad3a9f4a6a6906d1a78, State: Running, Role: LEADER
I20260926 09:27:22.309968 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8c68904d6a84ad3a9f4a6a6906d1a78 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:27:22.310030 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8c68904d6a84ad3a9f4a6a6906d1a78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c68904d6a84ad3a9f4a6a6906d1a78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37509 } }
I20260926 09:27:22.310318 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c68904d6a84ad3a9f4a6a6906d1a78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8c68904d6a84ad3a9f4a6a6906d1a78. Latest consensus state: current_term: 1 leader_uuid: "e8c68904d6a84ad3a9f4a6a6906d1a78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c68904d6a84ad3a9f4a6a6906d1a78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37509 } } }
I20260926 09:27:22.310329 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c68904d6a84ad3a9f4a6a6906d1a78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8c68904d6a84ad3a9f4a6a6906d1a78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c68904d6a84ad3a9f4a6a6906d1a78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37509 } } }
I20260926 09:27:22.310400 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c68904d6a84ad3a9f4a6a6906d1a78 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:22.310407 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c68904d6a84ad3a9f4a6a6906d1a78 [sys.catalog]: This master's current role is: LEADER
W20260926 09:27:22.311980 1019 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e8c68904d6a84ad3a9f4a6a6906d1a78: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:27:22.312228 1019 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:27:22.312299 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:27:22.312461 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 09:27:22 dist-test-slave-dfzm krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:22.313951 1010 catalog_manager.cc:1347] Generated new cluster ID: 4c5c67072bfe49d489824425f59d2d5f
I20260926 09:27:22.314005 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:27:22.317183 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:22.306530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59142 (local address 127.0.98.62:37509)
0926 09:27:22.306719 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:22.306727 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:22.306753 (+ 26us) server_negotiation.cc:407] Connection header received
0926 09:27:22.307200 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:22.307206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:22.307254 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:22.307379 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:22.308161 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.309448 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.310583 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.311195 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.311300 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:22.313960 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:22.313964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:22.313964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:22.313975 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:22.315703 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:22.316073 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:22.316077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:22.316077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:22.316171 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:22.316375 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:22.316376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:22.316376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:22.316611 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:22.316792 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:22.316867 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 09:27:22.317004 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
I20260926 09:27:22.346177 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:27:22.346776 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:27:22.351994 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8c68904d6a84ad3a9f4a6a6906d1a78: Generated new TSK 0
I20260926 09:27:22.352210 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:27:22.392072 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37509
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:22.486984 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:22.487193 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:22.487248 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:22.488353 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:22.488389 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:22.488415 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:27:22.489781 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37509
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:22.489954 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:22.490206 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:22.495059 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:22.495036 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:22.495025 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:22.495352 1024 server_base.cc:1034] running on GCE node
I20260926 09:27:22.495575 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:22.495810 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:22.496982 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790414842496956 us; error 35 us; skew 500 ppm
Sep 26 09:27:22 dist-test-slave-dfzm krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:22.499433 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:27:22.499898 1024 webserver.cc:460] Webserver started at http://127.0.98.1:41629/ using document root <none> and password file <none>
I20260926 09:27:22.500124 1024 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:22.500192 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:22.500334 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:22.501391 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "1aa834bb9712448f86cb24ce32003bca"
format_stamp: "Formatted at 2026-09-26 09:27:22 on dist-test-slave-dfzm"
server_key: "2d7aa9c45eaa9ebdce564ffab1a7657b"
server_key_iv: "43e6cde81da74208c6d97d70de1b2f0b"
server_key_version: "encryptionkey@0"
I20260926 09:27:22.501791 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1aa834bb9712448f86cb24ce32003bca"
format_stamp: "Formatted at 2026-09-26 09:27:22 on dist-test-slave-dfzm"
server_key: "2d7aa9c45eaa9ebdce564ffab1a7657b"
server_key_iv: "43e6cde81da74208c6d97d70de1b2f0b"
server_key_version: "encryptionkey@0"
I20260926 09:27:22.503115 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:27:22.503769 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:22.503930 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:27:22.504022 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "1aa834bb9712448f86cb24ce32003bca"
format_stamp: "Formatted at 2026-09-26 09:27:22 on dist-test-slave-dfzm"
server_key: "2d7aa9c45eaa9ebdce564ffab1a7657b"
server_key_iv: "43e6cde81da74208c6d97d70de1b2f0b"
server_key_version: "encryptionkey@0"
I20260926 09:27:22.504106 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:22.514648 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:22.516479 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:22.516649 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:22.516880 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:22.517264 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:22.517321 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:22.517360 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:22.517388 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:22.523242 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40385
I20260926 09:27:22.523291 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40385 every 8 connection(s)
I20260926 09:27:22.523626 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:27:22.527352 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 1024
I20260926 09:27:22.527462 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:27:22.527611 392 external_mini_cluster.cc:1428] Setting key 075083ee7480b497e47c65d09b8d4f51
Sep 26 09:27:22 dist-test-slave-dfzm krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:22.533923 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:22.524349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32999 (local address 127.0.98.62:37509)
0926 09:27:22.524409 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:22.524411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:22.524605 (+ 194us) server_negotiation.cc:407] Connection header received
0926 09:27:22.525145 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:22.525147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:22.525160 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:22.525211 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:22.526290 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.526825 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.528095 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.528382 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.528463 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:22.530754 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:22.530756 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:22.530756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:22.530763 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:22.532472 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:22.532873 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:22.532874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:22.532875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:22.532939 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:22.533172 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:22.533172 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:22.533173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:22.533341 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:22.533420 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:22.533706 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 09:27:22.533809 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 09:27:22.534008 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:22.524330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37509 (local address 127.0.98.1:32999)
0926 09:27:22.524487 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:22.524522 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:22.525031 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:22.525286 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:22.525289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:22.525930 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:22.526171 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:22.526175 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.526919 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.526920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:22.528019 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:22.528021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.528466 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.528466 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:22.528567 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:22.529129 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:22.529137 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:22.530668 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:22.532557 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:22.532563 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:22.532564 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:22.532795 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:22.533012 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:22.533012 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:22.533013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:22.533106 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:22.533412 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:22.533416 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:22.533528 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 09:27:22.533681 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 09:27:22.533821 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260926 09:27:22.534509 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37509
I20260926 09:27:22.534618 1153 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:22.534816 1153 heartbeater.cc:511] Master 127.0.98.62:37509 requested a full tablet report, sending...
I20260926 09:27:22.535341 962 ts_manager.cc:194] Registered new tserver with Master: 1aa834bb9712448f86cb24ce32003bca (127.0.98.1:40385)
I20260926 09:27:22.535956 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32999
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:27:22.552996 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37509
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:22.634084 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:22.634322 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:22.634361 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:22.635536 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:22.635622 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:22.635673 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:27:22.637117 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37509
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:22.637373 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:22.637637 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:22.642655 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:22.642629 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:22.642824 1158 server_base.cc:1034] running on GCE node
W20260926 09:27:22.642647 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:22.643177 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:22.643462 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:22.644631 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790414842644619 us; error 32 us; skew 500 ppm
Sep 26 09:27:22 dist-test-slave-dfzm krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:22.647259 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:27:22.647740 1158 webserver.cc:460] Webserver started at http://127.0.98.2:35459/ using document root <none> and password file <none>
I20260926 09:27:22.647982 1158 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:22.648029 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:22.648156 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:22.649215 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "215772e36424443a8fb7db031ab4e251"
format_stamp: "Formatted at 2026-09-26 09:27:22 on dist-test-slave-dfzm"
server_key: "176fb148b8cdf4908f4eaff589ba7ecd"
server_key_iv: "ab84a5e4f9ceb59e99af8ebd818f650f"
server_key_version: "encryptionkey@0"
I20260926 09:27:22.649567 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "215772e36424443a8fb7db031ab4e251"
format_stamp: "Formatted at 2026-09-26 09:27:22 on dist-test-slave-dfzm"
server_key: "176fb148b8cdf4908f4eaff589ba7ecd"
server_key_iv: "ab84a5e4f9ceb59e99af8ebd818f650f"
server_key_version: "encryptionkey@0"
I20260926 09:27:22.650913 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:22.651636 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:22.651819 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:22.651909 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "215772e36424443a8fb7db031ab4e251"
format_stamp: "Formatted at 2026-09-26 09:27:22 on dist-test-slave-dfzm"
server_key: "176fb148b8cdf4908f4eaff589ba7ecd"
server_key_iv: "ab84a5e4f9ceb59e99af8ebd818f650f"
server_key_version: "encryptionkey@0"
I20260926 09:27:22.651989 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:22.663321 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:22.665557 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:22.665794 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:22.666038 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:22.666384 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:22.666440 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:22.666481 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:22.666527 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:22.672385 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40487
I20260926 09:27:22.672430 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40487 every 8 connection(s)
I20260926 09:27:22.672749 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:27:22.677516 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 1158
I20260926 09:27:22.677639 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:27:22.677814 392 external_mini_cluster.cc:1428] Setting key 3d459b6292e7debaa56485dfa39054e7
Sep 26 09:27:22 dist-test-slave-dfzm krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:22.683368 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:22.673469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38239 (local address 127.0.98.62:37509)
0926 09:27:22.673552 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:22.673555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:22.673781 (+ 226us) server_negotiation.cc:407] Connection header received
0926 09:27:22.674306 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:22.674307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:22.674319 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:22.674371 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:22.675315 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.675817 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.677254 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.677568 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.677660 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:22.680186 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:22.680188 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:22.680188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:22.680197 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:22.681851 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:22.682326 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:22.682327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:22.682328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:22.682392 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:22.682621 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:22.682622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:22.682623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:22.682778 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:22.682853 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:22.683118 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 09:27:22.683219 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 09:27:22.683401 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:22.673430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37509 (local address 127.0.98.2:38239)
0926 09:27:22.673619 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:22.673664 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:22.674199 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:22.674445 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:22.674449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:22.675022 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:22.675215 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:22.675220 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.675913 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.675914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:22.677158 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:22.677166 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.677700 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.677702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:22.677859 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:22.678465 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:22.678476 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:22.680089 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:22.681941 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:22.681946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:22.681948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:22.682247 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:22.682461 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:22.682462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:22.682462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:22.682559 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:22.682845 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:22.682850 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:22.682971 (+ 121us) client_negotiation.cc:769] Sending connection context
0926 09:27:22.683056 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 09:27:22.683233 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260926 09:27:22.683931 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37509
I20260926 09:27:22.684046 1287 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:22.684254 1287 heartbeater.cc:511] Master 127.0.98.62:37509 requested a full tablet report, sending...
I20260926 09:27:22.684677 962 ts_manager.cc:194] Registered new tserver with Master: 215772e36424443a8fb7db031ab4e251 (127.0.98.2:40487)
I20260926 09:27:22.685073 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38239
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:27:22.703838 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37509
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:22.785175 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:22.785387 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:22.785413 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:22.786608 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:22.786652 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:22.786680 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:27:22.788133 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37509
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:22.788372 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:22.788618 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:22.793525 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:22.793558 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:22.793745 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:22.793754 1292 server_base.cc:1034] running on GCE node
I20260926 09:27:22.794016 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:22.794248 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:22.795423 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790414842795419 us; error 34 us; skew 500 ppm
Sep 26 09:27:22 dist-test-slave-dfzm krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:22.797884 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:27:22.798277 1292 webserver.cc:460] Webserver started at http://127.0.98.3:38521/ using document root <none> and password file <none>
I20260926 09:27:22.798578 1292 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:22.798666 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:22.798815 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:22.799891 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "a3e81a5145314beea583bb0d26e2a2be"
format_stamp: "Formatted at 2026-09-26 09:27:22 on dist-test-slave-dfzm"
server_key: "3a4536dd45f35b81e24b2196f6684b1c"
server_key_iv: "f0f968d2a4be2acfbbf21c0a2132597c"
server_key_version: "encryptionkey@0"
I20260926 09:27:22.800240 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a3e81a5145314beea583bb0d26e2a2be"
format_stamp: "Formatted at 2026-09-26 09:27:22 on dist-test-slave-dfzm"
server_key: "3a4536dd45f35b81e24b2196f6684b1c"
server_key_iv: "f0f968d2a4be2acfbbf21c0a2132597c"
server_key_version: "encryptionkey@0"
I20260926 09:27:22.801663 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:27:22.802407 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:22.802588 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:22.802670 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "a3e81a5145314beea583bb0d26e2a2be"
format_stamp: "Formatted at 2026-09-26 09:27:22 on dist-test-slave-dfzm"
server_key: "3a4536dd45f35b81e24b2196f6684b1c"
server_key_iv: "f0f968d2a4be2acfbbf21c0a2132597c"
server_key_version: "encryptionkey@0"
I20260926 09:27:22.802740 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:22.832321 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:22.834395 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:22.834555 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:22.834828 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:22.835181 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:22.835215 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:22.835237 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:22.835249 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:22.841238 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34857
I20260926 09:27:22.841315 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34857 every 8 connection(s)
I20260926 09:27:22.841699 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:27:22 dist-test-slave-dfzm krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:22.849145 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 1292
I20260926 09:27:22.849260 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:27:22.849432 392 external_mini_cluster.cc:1428] Setting key 106f1cf76fd971abc8610bbcdc426136
I20260926 09:27:22.851348 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:22.842470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60481 (local address 127.0.98.62:37509)
0926 09:27:22.843009 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:22.843011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:22.843023 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:27:22.843503 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:22.843504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:22.843517 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:22.843594 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:22.844486 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.844975 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.846029 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.846246 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.846306 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:22.848475 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:22.848476 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:22.848476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:22.848482 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:22.849974 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:22.850379 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:22.850380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:22.850381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:22.850443 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:22.850690 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:22.850691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:22.850691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:22.850809 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:22.850883 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:22.851171 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 09:27:22.851259 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507}
I20260926 09:27:22.851533 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:22.842448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37509 (local address 127.0.98.3:60481)
0926 09:27:22.842611 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:22.842659 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:22.843401 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:22.843656 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:22.843659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:22.844213 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:22.844408 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:22.844412 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.845051 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.845051 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:22.845950 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:22.845952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.846323 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.846323 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:22.846417 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:22.846897 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:22.846905 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:22.848391 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:22.850068 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:22.850074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:22.850077 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:22.850302 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:22.850509 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:22.850510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:22.850510 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:22.850622 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:22.850875 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:22.850881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:22.851031 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 09:27:22.851136 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 09:27:22.851299 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260926 09:27:22.852042 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37509
I20260926 09:27:22.852154 1421 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:22.852356 1421 heartbeater.cc:511] Master 127.0.98.62:37509 requested a full tablet report, sending...
I20260926 09:27:22.852774 962 ts_manager.cc:194] Registered new tserver with Master: a3e81a5145314beea583bb0d26e2a2be (127.0.98.3:34857)
I20260926 09:27:22.853101 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60481
I20260926 09:27:22.861142 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:27:22.868947 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:22.862832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59156 (local address 127.0.98.62:37509)
0926 09:27:22.862891 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:22.862892 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:22.863009 (+ 117us) server_negotiation.cc:407] Connection header received
0926 09:27:22.863051 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:22.863052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:22.863061 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:22.863105 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:22.863636 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.864248 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.864928 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.865155 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.865258 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:22.866441 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:22.866442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:22.866442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:22.866448 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:22.867955 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:22.868263 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:22.868265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:22.868265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:22.868324 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:22.868501 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:22.868501 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:22.868502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:22.868656 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:22.868733 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:22.868818 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 09:27:22.868880 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 09:27:22.871243 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:27:22.872822 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:27:22.879602 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:22.875371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53762 (local address 127.0.98.2:40487)
0926 09:27:22.875760 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:22.875764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:22.875779 (+ 15us) server_negotiation.cc:407] Connection header received
0926 09:27:22.875812 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:22.875816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:22.875832 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:22.876088 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:22.876166 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.877194 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.878390 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.879175 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.879259 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:22.879332 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:22.879480 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 09:27:22.879522 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":89,"threads_started":1}
I20260926 09:27:22.879637 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:22.875100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34857 (local address 127.0.0.1:44598)
0926 09:27:22.875218 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:22.875258 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:22.875328 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:22.875733 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:22.875736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:22.875739 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:22.875817 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:22.875819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.877117 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.877118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:22.878466 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:22.878468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.879290 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.879290 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:22.879360 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:22.879378 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 09:27:22.879457 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 09:27:22.879522 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":58,"thread_start_us":110,"threads_started":1}
I20260926 09:27:22.879879 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:22.875370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44598 (local address 127.0.98.3:34857)
0926 09:27:22.875528 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:22.875534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:22.875545 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:27:22.875585 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:22.875587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:22.875605 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:22.875702 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:22.875897 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.877019 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.878547 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.879222 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.879284 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:22.879326 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:22.879753 (+ 427us) server_negotiation.cc:299] Negotiation successful
0926 09:27:22.879804 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260926 09:27:22.879637 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:22.875104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40487 (local address 127.0.0.1:53762)
0926 09:27:22.875535 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:22.875543 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:22.875587 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:22.875953 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:22.875954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:22.875956 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:22.876004 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:22.876006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.877291 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.877292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:22.878304 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:22.878306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.879280 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.879282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:22.879361 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:22.879379 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 09:27:22.879457 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 09:27:22.879522 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":4,"thread_start_us":61,"threads_started":1}
I20260926 09:27:22.880527 1355 tablet_service.cc:1467] Processing CreateTablet for tablet f5745c9f4caf4f3abc93092f0230759b (DEFAULT_TABLE table=test-table [id=26fd4939bece4522bcb87a77cc0cd1a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:22.880702 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:22.875104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40385 (local address 127.0.0.1:37916)
0926 09:27:22.875377 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:22.875388 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:22.875442 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:22.876041 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:22.876042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:22.876044 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:22.876087 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:22.876089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.876976 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.876977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:22.878460 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:22.878463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.880442 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.880443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:22.880503 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:22.880508 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:27:22.880567 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 09:27:22.880611 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":69,"thread_start_us":89,"threads_started":1}
I20260926 09:27:22.880729 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:22.875563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37916 (local address 127.0.98.1:40385)
0926 09:27:22.875737 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:22.875742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:22.875751 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:27:22.875790 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:22.875793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:22.875816 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:22.875965 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:22.876166 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.876867 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.879367 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:22.880297 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:22.880360 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:22.880410 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:22.880606 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 09:27:22.880653 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
I20260926 09:27:22.880672 1221 tablet_service.cc:1467] Processing CreateTablet for tablet f5745c9f4caf4f3abc93092f0230759b (DEFAULT_TABLE table=test-table [id=26fd4939bece4522bcb87a77cc0cd1a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:22.880929 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5745c9f4caf4f3abc93092f0230759b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:22.881057 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5745c9f4caf4f3abc93092f0230759b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:22.881340 1087 tablet_service.cc:1467] Processing CreateTablet for tablet f5745c9f4caf4f3abc93092f0230759b (DEFAULT_TABLE table=test-table [id=26fd4939bece4522bcb87a77cc0cd1a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:22.881675 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5745c9f4caf4f3abc93092f0230759b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:22.883617 1438 tablet_bootstrap.cc:492] T f5745c9f4caf4f3abc93092f0230759b P a3e81a5145314beea583bb0d26e2a2be: Bootstrap starting.
I20260926 09:27:22.884284 1438 tablet_bootstrap.cc:654] T f5745c9f4caf4f3abc93092f0230759b P a3e81a5145314beea583bb0d26e2a2be: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:22.884534 1439 tablet_bootstrap.cc:492] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251: Bootstrap starting.
I20260926 09:27:22.884819 1440 tablet_bootstrap.cc:492] T f5745c9f4caf4f3abc93092f0230759b P 1aa834bb9712448f86cb24ce32003bca: Bootstrap starting.
I20260926 09:27:22.884833 1438 log.cc:824] T f5745c9f4caf4f3abc93092f0230759b P a3e81a5145314beea583bb0d26e2a2be: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:22.885222 1439 tablet_bootstrap.cc:654] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:22.885479 1440 tablet_bootstrap.cc:654] T f5745c9f4caf4f3abc93092f0230759b P 1aa834bb9712448f86cb24ce32003bca: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:22.885676 1438 tablet_bootstrap.cc:492] T f5745c9f4caf4f3abc93092f0230759b P a3e81a5145314beea583bb0d26e2a2be: No bootstrap required, opened a new log
I20260926 09:27:22.885747 1438 ts_tablet_manager.cc:1397] T f5745c9f4caf4f3abc93092f0230759b P a3e81a5145314beea583bb0d26e2a2be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:27:22.885833 1439 log.cc:824] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:22.886078 1440 log.cc:824] T f5745c9f4caf4f3abc93092f0230759b P 1aa834bb9712448f86cb24ce32003bca: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:22.886535 1439 tablet_bootstrap.cc:492] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251: No bootstrap required, opened a new log
I20260926 09:27:22.886626 1439 ts_tablet_manager.cc:1397] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:27:22.886861 1440 tablet_bootstrap.cc:492] T f5745c9f4caf4f3abc93092f0230759b P 1aa834bb9712448f86cb24ce32003bca: No bootstrap required, opened a new log
I20260926 09:27:22.886925 1440 ts_tablet_manager.cc:1397] T f5745c9f4caf4f3abc93092f0230759b P 1aa834bb9712448f86cb24ce32003bca: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 09:27:22.887866 1438 raft_consensus.cc:348] T f5745c9f4caf4f3abc93092f0230759b P a3e81a5145314beea583bb0d26e2a2be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e81a5145314beea583bb0d26e2a2be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34857 } } peers { permanent_uuid: "1aa834bb9712448f86cb24ce32003bca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "215772e36424443a8fb7db031ab4e251" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40487 } }
I20260926 09:27:22.888046 1438 raft_consensus.cc:374] T f5745c9f4caf4f3abc93092f0230759b P a3e81a5145314beea583bb0d26e2a2be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:22.888087 1438 raft_consensus.cc:729] T f5745c9f4caf4f3abc93092f0230759b P a3e81a5145314beea583bb0d26e2a2be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e81a5145314beea583bb0d26e2a2be, State: Initialized, Role: FOLLOWER
I20260926 09:27:22.888145 1439 raft_consensus.cc:348] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e81a5145314beea583bb0d26e2a2be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34857 } } peers { permanent_uuid: "1aa834bb9712448f86cb24ce32003bca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "215772e36424443a8fb7db031ab4e251" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40487 } }
I20260926 09:27:22.888334 1439 raft_consensus.cc:374] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:22.888376 1439 raft_consensus.cc:729] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 215772e36424443a8fb7db031ab4e251, State: Initialized, Role: FOLLOWER
I20260926 09:27:22.888499 1439 consensus_queue.cc:260] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e81a5145314beea583bb0d26e2a2be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34857 } } peers { permanent_uuid: "1aa834bb9712448f86cb24ce32003bca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "215772e36424443a8fb7db031ab4e251" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40487 } }
I20260926 09:27:22.888688 1438 consensus_queue.cc:260] T f5745c9f4caf4f3abc93092f0230759b P a3e81a5145314beea583bb0d26e2a2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e81a5145314beea583bb0d26e2a2be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34857 } } peers { permanent_uuid: "1aa834bb9712448f86cb24ce32003bca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "215772e36424443a8fb7db031ab4e251" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40487 } }
I20260926 09:27:22.888787 1439 ts_tablet_manager.cc:1428] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:27:22.888793 1287 heartbeater.cc:503] Master 127.0.98.62:37509 was elected leader, sending a full tablet report...
I20260926 09:27:22.889034 1438 ts_tablet_manager.cc:1428] T f5745c9f4caf4f3abc93092f0230759b P a3e81a5145314beea583bb0d26e2a2be: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:27:22.889060 1421 heartbeater.cc:503] Master 127.0.98.62:37509 was elected leader, sending a full tablet report...
I20260926 09:27:22.889276 1440 raft_consensus.cc:348] T f5745c9f4caf4f3abc93092f0230759b P 1aa834bb9712448f86cb24ce32003bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e81a5145314beea583bb0d26e2a2be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34857 } } peers { permanent_uuid: "1aa834bb9712448f86cb24ce32003bca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "215772e36424443a8fb7db031ab4e251" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40487 } }
I20260926 09:27:22.889575 1440 raft_consensus.cc:374] T f5745c9f4caf4f3abc93092f0230759b P 1aa834bb9712448f86cb24ce32003bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:22.889650 1440 raft_consensus.cc:729] T f5745c9f4caf4f3abc93092f0230759b P 1aa834bb9712448f86cb24ce32003bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aa834bb9712448f86cb24ce32003bca, State: Initialized, Role: FOLLOWER
I20260926 09:27:22.889772 1440 consensus_queue.cc:260] T f5745c9f4caf4f3abc93092f0230759b P 1aa834bb9712448f86cb24ce32003bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e81a5145314beea583bb0d26e2a2be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34857 } } peers { permanent_uuid: "1aa834bb9712448f86cb24ce32003bca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "215772e36424443a8fb7db031ab4e251" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40487 } }
I20260926 09:27:22.890089 1153 heartbeater.cc:503] Master 127.0.98.62:37509 was elected leader, sending a full tablet report...
I20260926 09:27:22.890349 1440 ts_tablet_manager.cc:1428] T f5745c9f4caf4f3abc93092f0230759b P 1aa834bb9712448f86cb24ce32003bca: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260926 09:27:22.923282 1288 tablet.cc:2369] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:27:23.014081 1444 raft_consensus.cc:482] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:27:23.014205 1444 raft_consensus.cc:504] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e81a5145314beea583bb0d26e2a2be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34857 } } peers { permanent_uuid: "1aa834bb9712448f86cb24ce32003bca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "215772e36424443a8fb7db031ab4e251" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40487 } }
I20260926 09:27:23.014564 1444 leader_election.cc:288] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3e81a5145314beea583bb0d26e2a2be (127.0.98.3:34857), 1aa834bb9712448f86cb24ce32003bca (127.0.98.1:40385)
I20260926 09:27:23.018539 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:23.014897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48555 (local address 127.0.98.1:40385)
0926 09:27:23.014988 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:23.014990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:23.015061 (+ 71us) server_negotiation.cc:407] Connection header received
0926 09:27:23.015167 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:23.015169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:23.015179 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:23.015252 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:23.015394 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:23.016096 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:23.017534 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:23.018158 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:23.018235 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:23.018247 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:23.018426 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 09:27:23.018464 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 09:27:23.018539 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:23.014883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46413 (local address 127.0.98.3:34857)
0926 09:27:23.014938 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:23.014940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:23.014951 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:27:23.014982 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:23.014983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:23.014995 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:23.015070 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:23.015244 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:23.016027 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:23.017533 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:23.018155 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:23.018235 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:23.018247 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:23.018426 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 09:27:23.018468 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:27:23.018561 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:23.014762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34857 (local address 127.0.98.2:46413)
0926 09:27:23.014843 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:23.014855 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:23.014902 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:23.015092 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:23.015093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:23.015095 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:23.015170 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:23.015172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:23.016111 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:23.016112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:23.017413 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:23.017417 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:23.018246 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:23.018247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:23.018350 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:23.018353 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:27:23.018403 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 09:27:23.018461 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":18}
I20260926 09:27:23.018567 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:23.014792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40385 (local address 127.0.98.2:48555)
0926 09:27:23.014976 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:23.014985 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:23.015062 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:23.015264 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:23.015265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:23.015266 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:23.015321 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:23.015322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:23.016174 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:23.016174 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:23.017413 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:23.017417 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:23.018246 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:23.018247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:23.018332 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:23.018339 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 09:27:23.018403 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 09:27:23.018461 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":33,"thread_start_us":101,"threads_started":1}
I20260926 09:27:23.018913 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5745c9f4caf4f3abc93092f0230759b" candidate_uuid: "215772e36424443a8fb7db031ab4e251" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e81a5145314beea583bb0d26e2a2be" is_pre_election: true
I20260926 09:27:23.019115 1375 raft_consensus.cc:2454] T f5745c9f4caf4f3abc93092f0230759b P a3e81a5145314beea583bb0d26e2a2be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 215772e36424443a8fb7db031ab4e251 in term 0.
I20260926 09:27:23.019313 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5745c9f4caf4f3abc93092f0230759b" candidate_uuid: "215772e36424443a8fb7db031ab4e251" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aa834bb9712448f86cb24ce32003bca" is_pre_election: true
I20260926 09:27:23.019363 1174 leader_election.cc:302] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 215772e36424443a8fb7db031ab4e251, a3e81a5145314beea583bb0d26e2a2be; no voters:
I20260926 09:27:23.019487 1107 raft_consensus.cc:2454] T f5745c9f4caf4f3abc93092f0230759b P 1aa834bb9712448f86cb24ce32003bca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 215772e36424443a8fb7db031ab4e251 in term 0.
I20260926 09:27:23.019495 1444 raft_consensus.cc:2781] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:27:23.019542 1444 raft_consensus.cc:482] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:27:23.019567 1444 raft_consensus.cc:3037] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:23.020285 1444 raft_consensus.cc:504] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e81a5145314beea583bb0d26e2a2be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34857 } } peers { permanent_uuid: "1aa834bb9712448f86cb24ce32003bca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "215772e36424443a8fb7db031ab4e251" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40487 } }
I20260926 09:27:23.020435 1444 leader_election.cc:288] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251 [CANDIDATE]: Term 1 election: Requested vote from peers a3e81a5145314beea583bb0d26e2a2be (127.0.98.3:34857), 1aa834bb9712448f86cb24ce32003bca (127.0.98.1:40385)
I20260926 09:27:23.020596 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5745c9f4caf4f3abc93092f0230759b" candidate_uuid: "215772e36424443a8fb7db031ab4e251" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e81a5145314beea583bb0d26e2a2be"
I20260926 09:27:23.020603 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5745c9f4caf4f3abc93092f0230759b" candidate_uuid: "215772e36424443a8fb7db031ab4e251" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aa834bb9712448f86cb24ce32003bca"
I20260926 09:27:23.020673 1375 raft_consensus.cc:3037] T f5745c9f4caf4f3abc93092f0230759b P a3e81a5145314beea583bb0d26e2a2be [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:23.020697 1107 raft_consensus.cc:3037] T f5745c9f4caf4f3abc93092f0230759b P 1aa834bb9712448f86cb24ce32003bca [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:23.021377 1107 raft_consensus.cc:2454] T f5745c9f4caf4f3abc93092f0230759b P 1aa834bb9712448f86cb24ce32003bca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 215772e36424443a8fb7db031ab4e251 in term 1.
I20260926 09:27:23.021418 1375 raft_consensus.cc:2454] T f5745c9f4caf4f3abc93092f0230759b P a3e81a5145314beea583bb0d26e2a2be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 215772e36424443a8fb7db031ab4e251 in term 1.
I20260926 09:27:23.021634 1175 leader_election.cc:302] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aa834bb9712448f86cb24ce32003bca, 215772e36424443a8fb7db031ab4e251; no voters:
I20260926 09:27:23.021747 1444 raft_consensus.cc:2781] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:23.021925 1444 raft_consensus.cc:686] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251 [term 1 LEADER]: Becoming Leader. State: Replica: 215772e36424443a8fb7db031ab4e251, State: Running, Role: LEADER
I20260926 09:27:23.022063 1444 consensus_queue.cc:237] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e81a5145314beea583bb0d26e2a2be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34857 } } peers { permanent_uuid: "1aa834bb9712448f86cb24ce32003bca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "215772e36424443a8fb7db031ab4e251" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40487 } }
I20260926 09:27:23.022956 961 catalog_manager.cc:5495] T f5745c9f4caf4f3abc93092f0230759b P 215772e36424443a8fb7db031ab4e251 reported cstate change: term changed from 0 to 1, leader changed from <none> to 215772e36424443a8fb7db031ab4e251 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "215772e36424443a8fb7db031ab4e251" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e81a5145314beea583bb0d26e2a2be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1aa834bb9712448f86cb24ce32003bca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "215772e36424443a8fb7db031ab4e251" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40487 } health_report { overall_health: HEALTHY } } }
W20260926 09:27:23.024307 1154 tablet.cc:2369] T f5745c9f4caf4f3abc93092f0230759b P 1aa834bb9712448f86cb24ce32003bca: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 09:27:23 dist-test-slave-dfzm krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 09:27:23.064064 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:23.056342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37922 (local address 127.0.98.1:40385)
0926 09:27:23.056398 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:23.056401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:23.056486 (+ 85us) server_negotiation.cc:407] Connection header received
0926 09:27:23.056544 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:23.056544 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:23.056557 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:23.056609 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:23.057188 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:23.057733 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:23.058449 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:23.058629 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:23.058682 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:23.061135 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:23.061145 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:23.061145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:23.061152 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:23.062985 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:23.063313 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:23.063317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:23.063317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:23.063384 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:23.063604 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:23.063605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:23.063605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:23.063755 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:23.063859 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:23.063930 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 09:27:23.063993 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Sep 26 09:27:23 dist-test-slave-dfzm krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414843, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 09:27:23 dist-test-slave-dfzm krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414843, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 09:27:23.087278 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:23.080293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37932 (local address 127.0.98.1:40385)
0926 09:27:23.080383 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:23.080385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:23.080412 (+ 27us) server_negotiation.cc:407] Connection header received
0926 09:27:23.080470 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:23.080471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:23.080480 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:23.080524 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:23.081100 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:23.081573 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:23.082366 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:23.082556 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:23.082612 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:23.084806 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:23.084807 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:23.084807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:23.084814 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:23.086188 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:23.086520 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:23.086521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:23.086521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:23.086585 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:23.086794 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:23.086794 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:23.086794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:23.086904 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:23.086974 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:23.087143 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 09:27:23.087205 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 09:27:23.092304 1422 tablet.cc:2369] T f5745c9f4caf4f3abc93092f0230759b P a3e81a5145314beea583bb0d26e2a2be: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 09:27:23 dist-test-slave-dfzm krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 09:27:23 dist-test-slave-dfzm krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 09:27:23.109767 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:23.102701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37936 (local address 127.0.98.1:40385)
0926 09:27:23.102775 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:23.102777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:23.102817 (+ 40us) server_negotiation.cc:407] Connection header received
0926 09:27:23.102881 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:23.102881 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:23.102891 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:23.102941 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:23.103439 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:23.103923 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:23.104598 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:23.104832 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:23.104886 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:23.107184 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:23.107186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:23.107186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:23.107194 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:23.108551 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:23.108869 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:23.108870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:23.108870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:23.108929 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:23.109128 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:23.109129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:23.109130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:23.109244 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:23.109431 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:23.109507 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 09:27:23.109570 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 09:27:23.110311 1067 tablet_service.cc:2908] Rejecting scan request for tablet f5745c9f4caf4f3abc93092f0230759b: Uninitialized: clean time has not yet been initialized
I20260926 09:27:23.111171 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 1024
I20260926 09:27:23.116122 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 1158
I20260926 09:27:23.120491 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 1292
I20260926 09:27:23.124788 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 930
2026-09-26T09:27:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2119 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:27:23 dist-test-slave-dfzm krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:27:23 dist-test-slave-dfzm krb5kdc[1462](info): set up 2 sockets
Sep 26 09:27:23 dist-test-slave-dfzm krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260926 09:27:24.181125 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:27:24 dist-test-slave-dfzm krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:27:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:27:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:27:24.298321 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43609
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:24.382365 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:24.382666 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:24.382714 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:24.384492 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:27:24.384552 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 09:27:24.384575 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:24.384598 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:24.384629 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:27:24.384658 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:27:24.386590 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43609
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:24.386806 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:24.386991 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:24.391738 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:24.391757 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:24.391754 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:24.391981 1475 server_base.cc:1034] running on GCE node
I20260926 09:27:24.392196 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:24.392450 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:24.393652 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790414844393636 us; error 47 us; skew 500 ppm
Sep 26 09:27:24 dist-test-slave-dfzm krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:24.400017 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:27:24.400555 1475 webserver.cc:460] Webserver started at http://127.0.98.62:35183/ using document root <none> and password file <none>
I20260926 09:27:24.400803 1475 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:24.400872 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:24.401034 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:24.402232 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "81904029a2ff42e988ef366158b0bb62"
format_stamp: "Formatted at 2026-09-26 09:27:24 on dist-test-slave-dfzm"
server_key: "daf0e8e9cc88b76033cec708e37c39b6"
server_key_iv: "8ccb3e47ad696a104b8bf40efd6b77cf"
server_key_version: "encryptionkey@0"
I20260926 09:27:24.402603 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "81904029a2ff42e988ef366158b0bb62"
format_stamp: "Formatted at 2026-09-26 09:27:24 on dist-test-slave-dfzm"
server_key: "daf0e8e9cc88b76033cec708e37c39b6"
server_key_iv: "8ccb3e47ad696a104b8bf40efd6b77cf"
server_key_version: "encryptionkey@0"
I20260926 09:27:24.403950 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:27:24.404723 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:24.404906 1475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:24.405012 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "81904029a2ff42e988ef366158b0bb62"
format_stamp: "Formatted at 2026-09-26 09:27:24 on dist-test-slave-dfzm"
server_key: "daf0e8e9cc88b76033cec708e37c39b6"
server_key_iv: "8ccb3e47ad696a104b8bf40efd6b77cf"
server_key_version: "encryptionkey@0"
I20260926 09:27:24.405088 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:24.416108 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:24.418191 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:24.418372 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:24.423239 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43609
I20260926 09:27:24.423287 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43609 every 8 connection(s)
I20260926 09:27:24.423646 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:27:24.424319 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:24.426687 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62: Bootstrap starting.
I20260926 09:27:24.427294 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:24.427593 1553 log.cc:824] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:24.428254 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62: No bootstrap required, opened a new log
I20260926 09:27:24.430411 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81904029a2ff42e988ef366158b0bb62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43609 } }
I20260926 09:27:24.430564 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:24.430589 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81904029a2ff42e988ef366158b0bb62, State: Initialized, Role: FOLLOWER
I20260926 09:27:24.430742 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81904029a2ff42e988ef366158b0bb62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43609 } }
I20260926 09:27:24.430814 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:27:24.430836 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:27:24.430887 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:24.431461 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81904029a2ff42e988ef366158b0bb62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43609 } }
I20260926 09:27:24.431553 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81904029a2ff42e988ef366158b0bb62; no voters:
I20260926 09:27:24.431770 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:27:24.431849 1556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:24.432009 1556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62 [term 1 LEADER]: Becoming Leader. State: Replica: 81904029a2ff42e988ef366158b0bb62, State: Running, Role: LEADER
I20260926 09:27:24.432030 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:27:24.432104 1556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81904029a2ff42e988ef366158b0bb62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43609 } }
I20260926 09:27:24.432478 1557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81904029a2ff42e988ef366158b0bb62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81904029a2ff42e988ef366158b0bb62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43609 } } }
I20260926 09:27:24.432515 1558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81904029a2ff42e988ef366158b0bb62. Latest consensus state: current_term: 1 leader_uuid: "81904029a2ff42e988ef366158b0bb62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81904029a2ff42e988ef366158b0bb62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43609 } } }
I20260926 09:27:24.432574 1557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:24.432588 1558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:24.432807 1564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:27:24.433319 1564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:27:24.433357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 1475
I20260926 09:27:24.433456 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:27:24.433670 392 external_mini_cluster.cc:1428] Setting key f0dac2c3e6a29d4a19e4ed22c956139c
I20260926 09:27:24.437791 1564 catalog_manager.cc:1347] Generated new cluster ID: 569b8047448e450489b167541e8540a2
I20260926 09:27:24.437857 1564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:27:24 dist-test-slave-dfzm krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:24.447810 1564 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:27:24.448366 1575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:24.434139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59172 (local address 127.0.98.62:43609)
0926 09:27:24.434290 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:24.434300 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:24.434333 (+ 33us) server_negotiation.cc:407] Connection header received
0926 09:27:24.434469 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:24.434476 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:24.434531 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:24.434682 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:24.435249 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.436648 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.437388 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.441999 (+ 4611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.442186 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:24.444634 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:24.444639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:24.444639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:24.444658 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:24.446591 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:24.446968 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:24.446971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:24.446972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:24.447067 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:24.447304 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:24.447305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:24.447306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:24.447621 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:24.447777 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:24.448023 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 09:27:24.448178 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260926 09:27:24.448495 1564 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:27:24.453869 1564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62: Generated new TSK 0
I20260926 09:27:24.454039 1564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:27:24.486603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43609
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:24.592905 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:24.593114 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:24.593139 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:24.594452 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:24.594516 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:24.594556 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:27:24.594612 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:27:24.596139 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:24.596405 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:24.596655 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:24.602535 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:24.602500 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:24.602494 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:24.602698 1579 server_base.cc:1034] running on GCE node
I20260926 09:27:24.602944 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:24.603207 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:24.604372 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790414844604365 us; error 37 us; skew 500 ppm
Sep 26 09:27:24 dist-test-slave-dfzm krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:24.606923 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:27:24.607304 1579 webserver.cc:460] Webserver started at http://127.0.98.1:42617/ using document root <none> and password file <none>
I20260926 09:27:24.607486 1579 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:24.607532 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:24.607618 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:24.608719 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "716891f111124d66909210ac2e206a72"
format_stamp: "Formatted at 2026-09-26 09:27:24 on dist-test-slave-dfzm"
server_key: "495c6c479b0981de30e1673b0b0ae2ea"
server_key_iv: "2ab8a68a79d7708e5380b5a37d6412f0"
server_key_version: "encryptionkey@0"
I20260926 09:27:24.609077 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "716891f111124d66909210ac2e206a72"
format_stamp: "Formatted at 2026-09-26 09:27:24 on dist-test-slave-dfzm"
server_key: "495c6c479b0981de30e1673b0b0ae2ea"
server_key_iv: "2ab8a68a79d7708e5380b5a37d6412f0"
server_key_version: "encryptionkey@0"
I20260926 09:27:24.610527 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:27:24.611265 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:24.611548 1579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:24.611636 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "716891f111124d66909210ac2e206a72"
format_stamp: "Formatted at 2026-09-26 09:27:24 on dist-test-slave-dfzm"
server_key: "495c6c479b0981de30e1673b0b0ae2ea"
server_key_iv: "2ab8a68a79d7708e5380b5a37d6412f0"
server_key_version: "encryptionkey@0"
I20260926 09:27:24.611693 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:24.636848 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:24.638830 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:24.638944 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:24.639389 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:24.639443 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:24.639475 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:24.639490 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:24.646574 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40879
I20260926 09:27:24.646644 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40879 every 8 connection(s)
Sep 26 09:27:24 dist-test-slave-dfzm krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:24.646943 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:27:24.649873 1575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:24.639955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57881 (local address 127.0.98.62:43609)
0926 09:27:24.640040 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:24.640042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:24.640195 (+ 153us) server_negotiation.cc:407] Connection header received
0926 09:27:24.640686 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:24.640688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:24.640705 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:24.640785 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:24.641703 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.642567 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.644186 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.644570 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.644652 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:24.647476 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:24.647478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:24.647478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:24.647486 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:24.648690 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:24.649053 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:24.649054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:24.649054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:24.649105 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:24.649309 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:24.649310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:24.649310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:24.649408 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:24.649460 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:24.649743 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 09:27:24.649811 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 09:27:24.650086 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:24.639885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43609 (local address 127.0.98.1:57881)
0926 09:27:24.640059 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:24.640100 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:24.640581 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:24.640883 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:24.640887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:24.641278 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:24.641570 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:24.641578 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.642666 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.642667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:24.644084 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:24.644086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.644671 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.644671 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:24.644842 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:24.645648 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:24.645659 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:24.647370 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:24.648760 (+ 1390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:24.648766 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:24.648768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:24.648985 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:24.649162 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:24.649162 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:24.649162 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:24.649257 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:24.649466 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:24.649470 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:24.649629 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 09:27:24.649727 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 09:27:24.649847 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260926 09:27:24.650592 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43609
I20260926 09:27:24.650676 1710 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:24.650857 1710 heartbeater.cc:511] Master 127.0.98.62:43609 requested a full tablet report, sending...
I20260926 09:27:24.651377 1507 ts_manager.cc:194] Registered new tserver with Master: 716891f111124d66909210ac2e206a72 (127.0.98.1:40879)
I20260926 09:27:24.652163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 1579
I20260926 09:27:24.652204 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57881
I20260926 09:27:24.652271 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:27:24.652456 392 external_mini_cluster.cc:1428] Setting key 6376466db123abf41acb4d112120c8c0
I20260926 09:27:24.657754 1575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:24.651723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54153 (local address 127.0.98.62:43609)
0926 09:27:24.651764 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:24.651766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:24.651833 (+ 67us) server_negotiation.cc:407] Connection header received
0926 09:27:24.651881 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:24.651882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:24.651890 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:24.651963 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:24.652363 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.652921 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.653974 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.654163 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.654214 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:24.655186 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:24.655188 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:24.655188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:24.655198 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:24.656610 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:24.656996 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:24.656997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:24.656997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:24.657055 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:24.657255 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:24.657256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:24.657256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:24.657343 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:24.657391 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:24.657621 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 09:27:24.657696 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:27:24.657852 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:24.651658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43609 (local address 127.0.98.1:54153)
0926 09:27:24.651775 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:24.651785 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:24.651841 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:24.651965 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:24.651965 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:24.652176 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:24.652262 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:24.652264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.653040 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.653040 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:24.653877 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:24.653879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.654250 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.654251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:24.654325 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:24.654788 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:24.654793 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:24.655099 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:24.656688 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:24.656689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:24.656690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:24.656919 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:24.657118 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:24.657119 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:24.657119 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:24.657192 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:24.657414 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:24.657415 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:24.657479 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 09:27:24.657562 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 09:27:24.657700 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:27:24.676662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43609
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:24.757472 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:24.757700 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:24.757725 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:24.758888 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:24.758932 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:24.758953 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:27:24.759011 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:27:24.760502 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:24.760736 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:24.760979 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:24.765892 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:24.765980 1719 server_base.cc:1034] running on GCE node
W20260926 09:27:24.765933 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:24.765906 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:24.766410 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:24.766701 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:24.767866 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790414844767846 us; error 35 us; skew 500 ppm
Sep 26 09:27:24 dist-test-slave-dfzm krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:24.770402 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:27:24.770936 1719 webserver.cc:460] Webserver started at http://127.0.98.2:34927/ using document root <none> and password file <none>
I20260926 09:27:24.771162 1719 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:24.771232 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:24.771380 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:24.772444 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "1844920c4227449fb5fb4248a7737a6d"
format_stamp: "Formatted at 2026-09-26 09:27:24 on dist-test-slave-dfzm"
server_key: "eda96683e056e9fbbd6770ae3f7eea45"
server_key_iv: "11d17fa94b12a6b9011db44d5757b2db"
server_key_version: "encryptionkey@0"
I20260926 09:27:24.772809 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1844920c4227449fb5fb4248a7737a6d"
format_stamp: "Formatted at 2026-09-26 09:27:24 on dist-test-slave-dfzm"
server_key: "eda96683e056e9fbbd6770ae3f7eea45"
server_key_iv: "11d17fa94b12a6b9011db44d5757b2db"
server_key_version: "encryptionkey@0"
I20260926 09:27:24.774307 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:27:24.775202 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:24.775437 1719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:24.775549 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "1844920c4227449fb5fb4248a7737a6d"
format_stamp: "Formatted at 2026-09-26 09:27:24 on dist-test-slave-dfzm"
server_key: "eda96683e056e9fbbd6770ae3f7eea45"
server_key_iv: "11d17fa94b12a6b9011db44d5757b2db"
server_key_version: "encryptionkey@0"
I20260926 09:27:24.775626 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:24.789139 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:24.791337 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:24.791509 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:24.792059 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:24.792124 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:24.792171 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:24.792217 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 09:27:24 dist-test-slave-dfzm krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:24.799528 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37457
I20260926 09:27:24.799681 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37457 every 8 connection(s)
I20260926 09:27:24.799949 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:27:24.801180 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 1719
I20260926 09:27:24.801275 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:27:24.801541 392 external_mini_cluster.cc:1428] Setting key c7834ca9ca7cc3d1974d5a841554c06f
I20260926 09:27:24.802479 1575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:24.792622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52825 (local address 127.0.98.62:43609)
0926 09:27:24.792698 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:24.792701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:24.792893 (+ 192us) server_negotiation.cc:407] Connection header received
0926 09:27:24.793378 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:24.793379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:24.793392 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:24.793476 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:24.794264 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.794777 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.795876 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.796094 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.796183 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:24.799203 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:24.799205 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:24.799205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:24.799212 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:24.800823 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:24.801343 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:24.801344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:24.801345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:24.801407 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:24.801728 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:24.801729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:24.801729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:24.801879 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:24.801957 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:24.802275 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 09:27:24.802361 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 09:27:24.802561 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:24.792605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43609 (local address 127.0.98.2:52825)
0926 09:27:24.792751 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:24.792792 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:24.793278 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:24.793529 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:24.793532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:24.793951 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:24.794156 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:24.794161 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.794903 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.794905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:24.795780 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:24.795783 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.796192 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.796194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:24.796331 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:24.796869 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:24.796877 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:24.799098 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:24.800931 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:24.800937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:24.800939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:24.801271 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:24.801478 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:24.801478 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:24.801479 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:24.801653 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:24.801948 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:24.801955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:24.802109 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 09:27:24.802226 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 09:27:24.802378 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260926 09:27:24.803013 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43609
I20260926 09:27:24.803174 1850 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:24.803395 1850 heartbeater.cc:511] Master 127.0.98.62:43609 requested a full tablet report, sending...
I20260926 09:27:24.803774 1507 ts_manager.cc:194] Registered new tserver with Master: 1844920c4227449fb5fb4248a7737a6d (127.0.98.2:37457)
I20260926 09:27:24.804248 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52825
I20260926 09:27:24.811174 1575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:24.804151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57527 (local address 127.0.98.62:43609)
0926 09:27:24.804375 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:24.804376 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:24.804384 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:27:24.804400 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:24.804401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:24.804409 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:24.804487 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:24.804894 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.805648 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.806790 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.807104 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.807148 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:24.808400 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:24.808401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:24.808402 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:24.808443 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:24.809906 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:24.810311 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:24.810311 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:24.810312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:24.810360 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:24.810602 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:24.810603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:24.810603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:24.810782 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:24.810848 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:24.811038 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 09:27:24.811109 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
I20260926 09:27:24.811241 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:24.804069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43609 (local address 127.0.98.2:57527)
0926 09:27:24.804197 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:24.804208 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:24.804261 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:24.804496 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:24.804497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:24.804729 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:24.804802 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:24.804804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.805763 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.805764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:24.806685 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:24.806688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.807231 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.807232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:24.807299 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:24.807808 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:24.807814 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:24.808280 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:24.809988 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:24.809989 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:24.809990 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:24.810214 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:24.810420 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:24.810420 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:24.810421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:24.810504 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:24.810872 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:24.810873 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:24.810926 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 09:27:24.811019 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 09:27:24.811133 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:27:24.829447 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43609
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:24.910705 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:24.910925 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:24.910949 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:24.912115 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:24.912160 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:24.912182 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:27:24.912225 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:27:24.913793 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:24.914037 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:24.914274 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:24.919267 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:24.919335 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:24.919456 1859 server_base.cc:1034] running on GCE node
W20260926 09:27:24.919570 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:24.919792 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:24.920053 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:24.921233 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790414844921219 us; error 30 us; skew 500 ppm
Sep 26 09:27:24 dist-test-slave-dfzm krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:24.923910 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:27:24.924299 1859 webserver.cc:460] Webserver started at http://127.0.98.3:40273/ using document root <none> and password file <none>
I20260926 09:27:24.924605 1859 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:24.924650 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:24.924741 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:24.925964 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "43c7c9e34d1f4663a3af7ba1267dd38a"
format_stamp: "Formatted at 2026-09-26 09:27:24 on dist-test-slave-dfzm"
server_key: "5471a81c1110cf112db5d0ae6890e919"
server_key_iv: "e5035d4e117abe1ee0833bde6e74ad5d"
server_key_version: "encryptionkey@0"
I20260926 09:27:24.926292 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "43c7c9e34d1f4663a3af7ba1267dd38a"
format_stamp: "Formatted at 2026-09-26 09:27:24 on dist-test-slave-dfzm"
server_key: "5471a81c1110cf112db5d0ae6890e919"
server_key_iv: "e5035d4e117abe1ee0833bde6e74ad5d"
server_key_version: "encryptionkey@0"
I20260926 09:27:24.927591 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 09:27:24.928382 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:24.928575 1859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:27:24.928660 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "43c7c9e34d1f4663a3af7ba1267dd38a"
format_stamp: "Formatted at 2026-09-26 09:27:24 on dist-test-slave-dfzm"
server_key: "5471a81c1110cf112db5d0ae6890e919"
server_key_iv: "e5035d4e117abe1ee0833bde6e74ad5d"
server_key_version: "encryptionkey@0"
I20260926 09:27:24.928714 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:24.938784 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:24.940706 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:24.940814 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:24.941256 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:24.941296 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:24.941318 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:24.941330 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:24.947996 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35357
Sep 26 09:27:24 dist-test-slave-dfzm krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:24.948083 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35357 every 8 connection(s)
I20260926 09:27:24.948657 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:27:24.951776 1575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:24.941879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47847 (local address 127.0.98.62:43609)
0926 09:27:24.941934 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:24.941937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:24.942141 (+ 204us) server_negotiation.cc:407] Connection header received
0926 09:27:24.942770 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:24.942771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:24.942786 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:24.942850 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:24.943698 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.944529 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.945682 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.945884 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.945979 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:24.948580 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:24.948582 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:24.948582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:24.948590 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:24.950405 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:24.950894 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:24.950895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:24.950896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:24.950955 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:24.951230 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:24.951231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:24.951231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:24.951342 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:24.951397 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:24.951647 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 09:27:24.951713 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:27:24.951924 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:24.941857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43609 (local address 127.0.98.3:47847)
0926 09:27:24.942012 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:24.942051 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:24.942666 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:24.942920 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:24.942926 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:24.943317 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:24.943589 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:24.943595 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.944658 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.944660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:24.945509 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:24.945511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.946000 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.946001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:24.946155 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:24.946725 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:24.946735 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:24.948479 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:24.950497 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:24.950503 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:24.950505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:24.950792 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:24.951037 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:24.951039 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:24.951039 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:24.951175 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:24.951401 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:24.951405 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:24.951516 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 09:27:24.951615 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 09:27:24.951736 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260926 09:27:24.952411 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43609
I20260926 09:27:24.952503 1990 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:24.952714 1990 heartbeater.cc:511] Master 127.0.98.62:43609 requested a full tablet report, sending...
I20260926 09:27:24.953107 1506 ts_manager.cc:194] Registered new tserver with Master: 43c7c9e34d1f4663a3af7ba1267dd38a (127.0.98.3:35357)
I20260926 09:27:24.953685 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47847
I20260926 09:27:24.954082 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 1859
I20260926 09:27:24.954178 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:27:24.954367 392 external_mini_cluster.cc:1428] Setting key 7e5b82363b3ae53b079ffa8442bac333
I20260926 09:27:24.955194 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:27:24.959645 1575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:24.953438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60421 (local address 127.0.98.62:43609)
0926 09:27:24.953641 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:24.953642 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:24.953649 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:27:24.953668 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:24.953668 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:24.953682 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:24.953736 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:24.954192 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.954836 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.955655 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.955951 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.956028 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:24.957059 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:24.957060 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:24.957060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:24.957066 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:24.958472 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:24.958897 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:24.958898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:24.958898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:24.958943 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:24.959167 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:24.959168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:24.959168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:24.959292 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:24.959344 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:24.959531 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 09:27:24.959586 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260926 09:27:24.959765 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:24.953347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43609 (local address 127.0.98.3:60421)
0926 09:27:24.953476 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:24.953487 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:24.953538 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:24.953757 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:24.953759 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:24.953991 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:24.954074 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:24.954076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.954932 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.954933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:24.955577 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:24.955579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.956030 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.956031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:24.956077 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:24.956534 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:24.956544 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:24.956942 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:24.958557 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:24.958559 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:24.958559 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:24.958810 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:24.959009 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:24.959009 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:24.959010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:24.959087 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:24.959363 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:24.959364 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:24.959418 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 09:27:24.959516 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 09:27:24.959628 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
Sep 26 09:27:24 dist-test-slave-dfzm krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414844, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 09:27:24 dist-test-slave-dfzm krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414844, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:24.988063 1575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:24.980552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59182 (local address 127.0.98.62:43609)
0926 09:27:24.980631 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:24.980633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:24.980642 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:27:24.980669 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:24.980670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:24.980682 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:24.980784 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:24.981469 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.982217 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.983151 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.983369 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.983461 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:24.985820 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:24.985822 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:24.985822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:24.985832 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:24.987097 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:24.987398 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:24.987399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:24.987399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:24.987476 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:24.987638 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:24.987639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:24.987639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:24.987739 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:24.987809 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:24.987894 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 09:27:24.987955 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:27:24.989883 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:27:24.991397 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:27:24.998576 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:24.993901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40879 (local address 127.0.0.1:52276)
0926 09:27:24.994195 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:24.994207 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:24.994266 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:24.994968 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:24.994969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:24.994971 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:24.995018 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:24.995020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.995980 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.995981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:24.997146 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:24.997158 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.998176 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.998176 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:24.998268 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:24.998427 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 09:27:24.998475 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 09:27:24.998505 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":83,"thread_start_us":99,"threads_started":1}
I20260926 09:27:24.998593 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:24.994082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54358 (local address 127.0.98.2:37457)
0926 09:27:24.994426 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:24.994430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:24.994440 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:27:24.994496 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:24.994498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:24.994517 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:24.994637 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:24.994802 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.995951 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.997428 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.998167 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.998241 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:24.998345 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:24.998443 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 09:27:24.998487 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":105,"threads_started":1}
I20260926 09:27:24.999034 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:24.993999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52276 (local address 127.0.98.1:40879)
0926 09:27:24.994780 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:24.994785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:24.994796 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:27:24.994846 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:24.994848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:24.994863 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:24.994970 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:24.995082 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.995884 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.997249 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.998086 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.998179 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:24.998268 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:24.998927 (+ 659us) server_negotiation.cc:299] Negotiation successful
0926 09:27:24.998962 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":80,"threads_started":1}
I20260926 09:27:24.998576 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:24.993886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37457 (local address 127.0.0.1:54358)
0926 09:27:24.994067 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:24.994109 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:24.994177 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:24.994641 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:24.994644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:24.994646 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:24.994712 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:24.994713 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.996075 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.996075 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:24.997349 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:24.997350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.998243 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.998244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:24.998308 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:24.998328 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 09:27:24.998416 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 09:27:24.998482 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":70,"thread_start_us":48,"threads_started":1}
I20260926 09:27:24.999696 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:24.994353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.0.98.3:35357)
0926 09:27:24.994521 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:24.994526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:24.994537 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:27:24.994595 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:24.994597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:24.994614 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:24.994689 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:24.994875 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.995964 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.998382 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.999207 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.999288 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:24.999375 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:24.999592 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 09:27:24.999627 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
I20260926 09:27:24.999688 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:24.994078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35357 (local address 127.0.0.1:56872)
0926 09:27:24.994193 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:24.994201 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:24.994240 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:24.994752 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:24.994753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:24.994755 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:24.994797 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:24.994798 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.996049 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.996050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:24.997334 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:24.997337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:24.999408 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:24.999409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:24.999485 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:24.999490 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:27:24.999563 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 09:27:24.999606 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":2,"thread_start_us":69,"threads_started":1}
I20260926 09:27:24.999961 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 236ed5ca185747a09b7bfe38189c2e7b (DEFAULT_TABLE table=test-table [id=e4f648b6f118496d860f1d7aa8be8652]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:25.000356 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 236ed5ca185747a09b7bfe38189c2e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:25.003726 2013 tablet_bootstrap.cc:492] T 236ed5ca185747a09b7bfe38189c2e7b P 1844920c4227449fb5fb4248a7737a6d: Bootstrap starting.
I20260926 09:27:25.004135 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 236ed5ca185747a09b7bfe38189c2e7b (DEFAULT_TABLE table=test-table [id=e4f648b6f118496d860f1d7aa8be8652]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:25.004379 2013 tablet_bootstrap.cc:654] T 236ed5ca185747a09b7bfe38189c2e7b P 1844920c4227449fb5fb4248a7737a6d: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:25.004529 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 236ed5ca185747a09b7bfe38189c2e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:25.004714 2013 log.cc:824] T 236ed5ca185747a09b7bfe38189c2e7b P 1844920c4227449fb5fb4248a7737a6d: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:25.005496 2013 tablet_bootstrap.cc:492] T 236ed5ca185747a09b7bfe38189c2e7b P 1844920c4227449fb5fb4248a7737a6d: No bootstrap required, opened a new log
I20260926 09:27:25.005609 2013 ts_tablet_manager.cc:1397] T 236ed5ca185747a09b7bfe38189c2e7b P 1844920c4227449fb5fb4248a7737a6d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:27:25.006234 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 236ed5ca185747a09b7bfe38189c2e7b (DEFAULT_TABLE table=test-table [id=e4f648b6f118496d860f1d7aa8be8652]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:25.006604 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 236ed5ca185747a09b7bfe38189c2e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:25.006999 2013 raft_consensus.cc:348] T 236ed5ca185747a09b7bfe38189c2e7b P 1844920c4227449fb5fb4248a7737a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.007122 2013 raft_consensus.cc:374] T 236ed5ca185747a09b7bfe38189c2e7b P 1844920c4227449fb5fb4248a7737a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:25.007169 2013 raft_consensus.cc:729] T 236ed5ca185747a09b7bfe38189c2e7b P 1844920c4227449fb5fb4248a7737a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1844920c4227449fb5fb4248a7737a6d, State: Initialized, Role: FOLLOWER
I20260926 09:27:25.007288 2013 consensus_queue.cc:260] T 236ed5ca185747a09b7bfe38189c2e7b P 1844920c4227449fb5fb4248a7737a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.007566 2013 ts_tablet_manager.cc:1428] T 236ed5ca185747a09b7bfe38189c2e7b P 1844920c4227449fb5fb4248a7737a6d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:27:25.007586 1850 heartbeater.cc:503] Master 127.0.98.62:43609 was elected leader, sending a full tablet report...
I20260926 09:27:25.008441 2016 tablet_bootstrap.cc:492] T 236ed5ca185747a09b7bfe38189c2e7b P 43c7c9e34d1f4663a3af7ba1267dd38a: Bootstrap starting.
I20260926 09:27:25.009083 2016 tablet_bootstrap.cc:654] T 236ed5ca185747a09b7bfe38189c2e7b P 43c7c9e34d1f4663a3af7ba1267dd38a: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:25.009311 2016 log.cc:824] T 236ed5ca185747a09b7bfe38189c2e7b P 43c7c9e34d1f4663a3af7ba1267dd38a: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:25.010150 2016 tablet_bootstrap.cc:492] T 236ed5ca185747a09b7bfe38189c2e7b P 43c7c9e34d1f4663a3af7ba1267dd38a: No bootstrap required, opened a new log
I20260926 09:27:25.010233 2016 ts_tablet_manager.cc:1397] T 236ed5ca185747a09b7bfe38189c2e7b P 43c7c9e34d1f4663a3af7ba1267dd38a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:27:25.011224 2018 tablet_bootstrap.cc:492] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72: Bootstrap starting.
I20260926 09:27:25.011833 2018 tablet_bootstrap.cc:654] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:25.012130 2018 log.cc:824] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:25.012148 2016 raft_consensus.cc:348] T 236ed5ca185747a09b7bfe38189c2e7b P 43c7c9e34d1f4663a3af7ba1267dd38a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.012336 2016 raft_consensus.cc:374] T 236ed5ca185747a09b7bfe38189c2e7b P 43c7c9e34d1f4663a3af7ba1267dd38a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:25.012374 2016 raft_consensus.cc:729] T 236ed5ca185747a09b7bfe38189c2e7b P 43c7c9e34d1f4663a3af7ba1267dd38a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c7c9e34d1f4663a3af7ba1267dd38a, State: Initialized, Role: FOLLOWER
I20260926 09:27:25.012506 2016 consensus_queue.cc:260] T 236ed5ca185747a09b7bfe38189c2e7b P 43c7c9e34d1f4663a3af7ba1267dd38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.012810 2016 ts_tablet_manager.cc:1428] T 236ed5ca185747a09b7bfe38189c2e7b P 43c7c9e34d1f4663a3af7ba1267dd38a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:27:25.012841 1990 heartbeater.cc:503] Master 127.0.98.62:43609 was elected leader, sending a full tablet report...
I20260926 09:27:25.012959 2018 tablet_bootstrap.cc:492] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72: No bootstrap required, opened a new log
I20260926 09:27:25.013024 2018 ts_tablet_manager.cc:1397] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:27:25.015424 2018 raft_consensus.cc:348] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.015614 2018 raft_consensus.cc:374] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:25.015640 2018 raft_consensus.cc:729] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716891f111124d66909210ac2e206a72, State: Initialized, Role: FOLLOWER
I20260926 09:27:25.015743 2018 consensus_queue.cc:260] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.015971 2018 ts_tablet_manager.cc:1428] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 09:27:25.016034 1710 heartbeater.cc:503] Master 127.0.98.62:43609 was elected leader, sending a full tablet report...
W20260926 09:27:25.050563 1851 tablet.cc:2369] T 236ed5ca185747a09b7bfe38189c2e7b P 1844920c4227449fb5fb4248a7737a6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:27:25.053040 2021 raft_consensus.cc:482] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:27:25.053189 2021 raft_consensus.cc:504] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.053613 2021 leader_election.cc:288] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1844920c4227449fb5fb4248a7737a6d (127.0.98.2:37457), 43c7c9e34d1f4663a3af7ba1267dd38a (127.0.98.3:35357)
I20260926 09:27:25.056579 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:25.054145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40419 (local address 127.0.98.3:35357)
0926 09:27:25.054202 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:25.054204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:25.054212 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:27:25.054241 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:25.054242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:25.054254 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:25.054320 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:25.054468 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.054935 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.055862 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.056222 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.056276 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:25.056283 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:25.056458 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 09:27:25.056506 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:27:25.056617 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:25.053924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44075 (local address 127.0.98.2:37457)
0926 09:27:25.053998 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:25.054000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:25.054012 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:27:25.054103 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:25.054104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:25.054118 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:25.054179 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:25.054343 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.054884 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.055892 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.056277 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.056349 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:25.056358 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:25.056521 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 09:27:25.056557 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 09:27:25.056608 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:25.053977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35357 (local address 127.0.98.1:40419)
0926 09:27:25.054154 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:25.054164 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:25.054200 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:25.054331 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:25.054332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:25.054334 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:25.054402 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:25.054403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.055026 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.055027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:25.055768 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:25.055770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.056303 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.056304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:25.056382 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:25.056386 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:27:25.056444 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 09:27:25.056474 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":2,"thread_start_us":103,"threads_started":1}
I20260926 09:27:25.056638 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:25.053845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37457 (local address 127.0.98.1:44075)
0926 09:27:25.053916 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:25.053930 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:25.054010 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:25.054196 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:25.054197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:25.054199 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:25.054274 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:25.054276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.054958 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.054959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:25.055768 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:25.055770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.056360 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.056361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:25.056444 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:25.056448 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:27:25.056503 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 09:27:25.056538 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":11}
I20260926 09:27:25.057114 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "236ed5ca185747a09b7bfe38189c2e7b" candidate_uuid: "716891f111124d66909210ac2e206a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" is_pre_election: true
I20260926 09:27:25.057114 1799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "236ed5ca185747a09b7bfe38189c2e7b" candidate_uuid: "716891f111124d66909210ac2e206a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1844920c4227449fb5fb4248a7737a6d" is_pre_election: true
I20260926 09:27:25.057263 1944 raft_consensus.cc:2454] T 236ed5ca185747a09b7bfe38189c2e7b P 43c7c9e34d1f4663a3af7ba1267dd38a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716891f111124d66909210ac2e206a72 in term 0.
I20260926 09:27:25.057263 1799 raft_consensus.cc:2454] T 236ed5ca185747a09b7bfe38189c2e7b P 1844920c4227449fb5fb4248a7737a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716891f111124d66909210ac2e206a72 in term 0.
I20260926 09:27:25.057495 1595 leader_election.cc:302] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1844920c4227449fb5fb4248a7737a6d, 716891f111124d66909210ac2e206a72; no voters:
I20260926 09:27:25.057651 2021 raft_consensus.cc:2781] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:27:25.057721 2021 raft_consensus.cc:482] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:27:25.057739 2021 raft_consensus.cc:3037] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:25.058703 2021 raft_consensus.cc:504] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.058822 2021 leader_election.cc:288] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72 [CANDIDATE]: Term 1 election: Requested vote from peers 1844920c4227449fb5fb4248a7737a6d (127.0.98.2:37457), 43c7c9e34d1f4663a3af7ba1267dd38a (127.0.98.3:35357)
I20260926 09:27:25.059010 1799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "236ed5ca185747a09b7bfe38189c2e7b" candidate_uuid: "716891f111124d66909210ac2e206a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1844920c4227449fb5fb4248a7737a6d"
I20260926 09:27:25.059026 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "236ed5ca185747a09b7bfe38189c2e7b" candidate_uuid: "716891f111124d66909210ac2e206a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a"
I20260926 09:27:25.059084 1799 raft_consensus.cc:3037] T 236ed5ca185747a09b7bfe38189c2e7b P 1844920c4227449fb5fb4248a7737a6d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:25.059134 1944 raft_consensus.cc:3037] T 236ed5ca185747a09b7bfe38189c2e7b P 43c7c9e34d1f4663a3af7ba1267dd38a [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:25.059790 1799 raft_consensus.cc:2454] T 236ed5ca185747a09b7bfe38189c2e7b P 1844920c4227449fb5fb4248a7737a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716891f111124d66909210ac2e206a72 in term 1.
I20260926 09:27:25.059890 1944 raft_consensus.cc:2454] T 236ed5ca185747a09b7bfe38189c2e7b P 43c7c9e34d1f4663a3af7ba1267dd38a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716891f111124d66909210ac2e206a72 in term 1.
I20260926 09:27:25.059947 1595 leader_election.cc:302] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1844920c4227449fb5fb4248a7737a6d, 716891f111124d66909210ac2e206a72; no voters:
I20260926 09:27:25.060067 2021 raft_consensus.cc:2781] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:25.060252 2021 raft_consensus.cc:686] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72 [term 1 LEADER]: Becoming Leader. State: Replica: 716891f111124d66909210ac2e206a72, State: Running, Role: LEADER
I20260926 09:27:25.060360 2021 consensus_queue.cc:237] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.061264 1506 catalog_manager.cc:5495] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 716891f111124d66909210ac2e206a72 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "716891f111124d66909210ac2e206a72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } health_report { overall_health: UNKNOWN } } }
I20260926 09:27:25.073089 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:25.069990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52280 (local address 127.0.98.1:40879)
0926 09:27:25.070044 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:25.070045 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:25.070061 (+ 16us) server_negotiation.cc:407] Connection header received
0926 09:27:25.070145 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:25.070146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:25.070175 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:25.070265 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:27:25.070559 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.071365 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.072245 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.072466 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.072525 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:25.072748 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:27:25.072840 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:27:25.072913 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:25.073012 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 09:27:25.073033 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 09:27:25.074982 1799 raft_consensus.cc:1261] T 236ed5ca185747a09b7bfe38189c2e7b P 1844920c4227449fb5fb4248a7737a6d [term 1 FOLLOWER]: Refusing update from remote peer 716891f111124d66909210ac2e206a72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:25.074982 1944 raft_consensus.cc:1261] T 236ed5ca185747a09b7bfe38189c2e7b P 43c7c9e34d1f4663a3af7ba1267dd38a [term 1 FOLLOWER]: Refusing update from remote peer 716891f111124d66909210ac2e206a72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:25.075292 2021 consensus_queue.cc:1035] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:25.075424 2023 consensus_queue.cc:1035] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:25.078106 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333539205423869952 to 7333539205367779328. Current Snapshot: MvccSnapshot[applied={T|T < 7333539205423869952}]
I20260926 09:27:25.078224 2026 mvcc.cc:205] Tried to move back new op lower bound from 7333539205423869952 to 7333539205367779328. Current Snapshot: MvccSnapshot[applied={T|T < 7333539205423869952 or (T in {7333539205423869952})}]
I20260926 09:27:25.081465 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:25.081573 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:25.082320 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62: Sending DeleteTablet for 3 replicas of tablet 236ed5ca185747a09b7bfe38189c2e7b
I20260926 09:27:25.082653 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 236ed5ca185747a09b7bfe38189c2e7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54358
I20260926 09:27:25.082683 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 236ed5ca185747a09b7bfe38189c2e7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52276
I20260926 09:27:25.082885 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 236ed5ca185747a09b7bfe38189c2e7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56872
I20260926 09:27:25.082933 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:27:25.083089 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:27:25.083271 2038 tablet_replica.cc:330] T 236ed5ca185747a09b7bfe38189c2e7b P 43c7c9e34d1f4663a3af7ba1267dd38a: stopping tablet replica
I20260926 09:27:25.083310 2036 tablet_replica.cc:330] T 236ed5ca185747a09b7bfe38189c2e7b P 1844920c4227449fb5fb4248a7737a6d: stopping tablet replica
I20260926 09:27:25.083411 2036 raft_consensus.cc:2229] T 236ed5ca185747a09b7bfe38189c2e7b P 1844920c4227449fb5fb4248a7737a6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:25.083506 2038 raft_consensus.cc:2229] T 236ed5ca185747a09b7bfe38189c2e7b P 43c7c9e34d1f4663a3af7ba1267dd38a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:25.083535 2036 raft_consensus.cc:2258] T 236ed5ca185747a09b7bfe38189c2e7b P 1844920c4227449fb5fb4248a7737a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:25.083698 2037 tablet_replica.cc:330] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72: stopping tablet replica
I20260926 09:27:25.083786 2037 raft_consensus.cc:2229] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:27:25.083882 2037 raft_consensus.cc:2258] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:25.084061 2038 raft_consensus.cc:2258] T 236ed5ca185747a09b7bfe38189c2e7b P 43c7c9e34d1f4663a3af7ba1267dd38a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:25.084321 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 5d775f1a16be49c2811fb688dfb82c67 (DEFAULT_TABLE table=test-table [id=958c7dcedb9a485e9be2b96180b62e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:25.084321 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 5d775f1a16be49c2811fb688dfb82c67 (DEFAULT_TABLE table=test-table [id=958c7dcedb9a485e9be2b96180b62e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:25.084443 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d775f1a16be49c2811fb688dfb82c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:25.084452 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d775f1a16be49c2811fb688dfb82c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:25.084692 2038 ts_tablet_manager.cc:1905] T 236ed5ca185747a09b7bfe38189c2e7b P 43c7c9e34d1f4663a3af7ba1267dd38a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:25.085778 2013 tablet_bootstrap.cc:492] T 5d775f1a16be49c2811fb688dfb82c67 P 1844920c4227449fb5fb4248a7737a6d: Bootstrap starting.
I20260926 09:27:25.085816 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 5d775f1a16be49c2811fb688dfb82c67 (DEFAULT_TABLE table=test-table [id=958c7dcedb9a485e9be2b96180b62e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:25.085914 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d775f1a16be49c2811fb688dfb82c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:25.086194 2013 tablet_bootstrap.cc:654] T 5d775f1a16be49c2811fb688dfb82c67 P 1844920c4227449fb5fb4248a7737a6d: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:25.086633 2038 ts_tablet_manager.cc:1918] T 236ed5ca185747a09b7bfe38189c2e7b P 43c7c9e34d1f4663a3af7ba1267dd38a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:27:25.086753 2038 log.cc:1196] T 236ed5ca185747a09b7bfe38189c2e7b P 43c7c9e34d1f4663a3af7ba1267dd38a: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/236ed5ca185747a09b7bfe38189c2e7b
I20260926 09:27:25.087086 2038 ts_tablet_manager.cc:1939] T 236ed5ca185747a09b7bfe38189c2e7b P 43c7c9e34d1f4663a3af7ba1267dd38a: Deleting consensus metadata
I20260926 09:27:25.087563 1491 catalog_manager.cc:4841] TS 43c7c9e34d1f4663a3af7ba1267dd38a (127.0.98.3:35357): tablet 236ed5ca185747a09b7bfe38189c2e7b (table test-table [id=e4f648b6f118496d860f1d7aa8be8652]) successfully deleted
I20260926 09:27:25.087736 2018 tablet_bootstrap.cc:492] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72: Bootstrap starting.
I20260926 09:27:25.087773 2013 tablet_bootstrap.cc:492] T 5d775f1a16be49c2811fb688dfb82c67 P 1844920c4227449fb5fb4248a7737a6d: No bootstrap required, opened a new log
I20260926 09:27:25.087809 2013 ts_tablet_manager.cc:1397] T 5d775f1a16be49c2811fb688dfb82c67 P 1844920c4227449fb5fb4248a7737a6d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:27:25.087972 2013 raft_consensus.cc:348] T 5d775f1a16be49c2811fb688dfb82c67 P 1844920c4227449fb5fb4248a7737a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.088060 2013 raft_consensus.cc:374] T 5d775f1a16be49c2811fb688dfb82c67 P 1844920c4227449fb5fb4248a7737a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:25.088088 2013 raft_consensus.cc:729] T 5d775f1a16be49c2811fb688dfb82c67 P 1844920c4227449fb5fb4248a7737a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1844920c4227449fb5fb4248a7737a6d, State: Initialized, Role: FOLLOWER
I20260926 09:27:25.088166 2018 tablet_bootstrap.cc:654] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:25.088141 2013 consensus_queue.cc:260] T 5d775f1a16be49c2811fb688dfb82c67 P 1844920c4227449fb5fb4248a7737a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.088234 2013 ts_tablet_manager.cc:1428] T 5d775f1a16be49c2811fb688dfb82c67 P 1844920c4227449fb5fb4248a7737a6d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:25.088330 2016 tablet_bootstrap.cc:492] T 5d775f1a16be49c2811fb688dfb82c67 P 43c7c9e34d1f4663a3af7ba1267dd38a: Bootstrap starting.
I20260926 09:27:25.088732 2016 tablet_bootstrap.cc:654] T 5d775f1a16be49c2811fb688dfb82c67 P 43c7c9e34d1f4663a3af7ba1267dd38a: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:25.088833 2037 ts_tablet_manager.cc:1905] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:25.089339 2016 tablet_bootstrap.cc:492] T 5d775f1a16be49c2811fb688dfb82c67 P 43c7c9e34d1f4663a3af7ba1267dd38a: No bootstrap required, opened a new log
I20260926 09:27:25.089385 2016 ts_tablet_manager.cc:1397] T 5d775f1a16be49c2811fb688dfb82c67 P 43c7c9e34d1f4663a3af7ba1267dd38a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:25.089524 2016 raft_consensus.cc:348] T 5d775f1a16be49c2811fb688dfb82c67 P 43c7c9e34d1f4663a3af7ba1267dd38a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.089620 2016 raft_consensus.cc:374] T 5d775f1a16be49c2811fb688dfb82c67 P 43c7c9e34d1f4663a3af7ba1267dd38a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:25.089646 2016 raft_consensus.cc:729] T 5d775f1a16be49c2811fb688dfb82c67 P 43c7c9e34d1f4663a3af7ba1267dd38a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c7c9e34d1f4663a3af7ba1267dd38a, State: Initialized, Role: FOLLOWER
I20260926 09:27:25.089695 2016 consensus_queue.cc:260] T 5d775f1a16be49c2811fb688dfb82c67 P 43c7c9e34d1f4663a3af7ba1267dd38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.089781 2016 ts_tablet_manager.cc:1428] T 5d775f1a16be49c2811fb688dfb82c67 P 43c7c9e34d1f4663a3af7ba1267dd38a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:25.089924 2018 tablet_bootstrap.cc:492] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72: No bootstrap required, opened a new log
I20260926 09:27:25.089991 2018 ts_tablet_manager.cc:1397] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 09:27:25.089998 2036 ts_tablet_manager.cc:1905] T 236ed5ca185747a09b7bfe38189c2e7b P 1844920c4227449fb5fb4248a7737a6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:25.090139 2018 raft_consensus.cc:348] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.090229 2018 raft_consensus.cc:374] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:25.090251 2018 raft_consensus.cc:729] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716891f111124d66909210ac2e206a72, State: Initialized, Role: FOLLOWER
I20260926 09:27:25.090304 2018 consensus_queue.cc:260] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.090408 2018 ts_tablet_manager.cc:1428] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 09:27:25.091483 2037 ts_tablet_manager.cc:1918] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:27:25.091539 2037 log.cc:1196] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-0/wal/wals/236ed5ca185747a09b7bfe38189c2e7b
I20260926 09:27:25.091701 2036 ts_tablet_manager.cc:1918] T 236ed5ca185747a09b7bfe38189c2e7b P 1844920c4227449fb5fb4248a7737a6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:27:25.091784 2036 log.cc:1196] T 236ed5ca185747a09b7bfe38189c2e7b P 1844920c4227449fb5fb4248a7737a6d: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-1/wal/wals/236ed5ca185747a09b7bfe38189c2e7b
I20260926 09:27:25.091820 2037 ts_tablet_manager.cc:1939] T 236ed5ca185747a09b7bfe38189c2e7b P 716891f111124d66909210ac2e206a72: Deleting consensus metadata
I20260926 09:27:25.092031 2036 ts_tablet_manager.cc:1939] T 236ed5ca185747a09b7bfe38189c2e7b P 1844920c4227449fb5fb4248a7737a6d: Deleting consensus metadata
I20260926 09:27:25.092196 1493 catalog_manager.cc:4841] TS 716891f111124d66909210ac2e206a72 (127.0.98.1:40879): tablet 236ed5ca185747a09b7bfe38189c2e7b (table test-table [id=e4f648b6f118496d860f1d7aa8be8652]) successfully deleted
I20260926 09:27:25.092451 1491 catalog_manager.cc:4841] TS 1844920c4227449fb5fb4248a7737a6d (127.0.98.2:37457): tablet 236ed5ca185747a09b7bfe38189c2e7b (table test-table [id=e4f648b6f118496d860f1d7aa8be8652]) successfully deleted
W20260926 09:27:25.147781 1711 tablet.cc:2369] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:27:25.192886 2023 raft_consensus.cc:482] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:27:25.192994 2023 raft_consensus.cc:504] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.193140 2023 leader_election.cc:288] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1844920c4227449fb5fb4248a7737a6d (127.0.98.2:37457), 43c7c9e34d1f4663a3af7ba1267dd38a (127.0.98.3:35357)
I20260926 09:27:25.193388 1799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d775f1a16be49c2811fb688dfb82c67" candidate_uuid: "716891f111124d66909210ac2e206a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1844920c4227449fb5fb4248a7737a6d" is_pre_election: true
I20260926 09:27:25.193421 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d775f1a16be49c2811fb688dfb82c67" candidate_uuid: "716891f111124d66909210ac2e206a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" is_pre_election: true
I20260926 09:27:25.193514 1944 raft_consensus.cc:2454] T 5d775f1a16be49c2811fb688dfb82c67 P 43c7c9e34d1f4663a3af7ba1267dd38a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716891f111124d66909210ac2e206a72 in term 0.
I20260926 09:27:25.193517 1799 raft_consensus.cc:2454] T 5d775f1a16be49c2811fb688dfb82c67 P 1844920c4227449fb5fb4248a7737a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716891f111124d66909210ac2e206a72 in term 0.
I20260926 09:27:25.193676 1595 leader_election.cc:302] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43c7c9e34d1f4663a3af7ba1267dd38a, 716891f111124d66909210ac2e206a72; no voters:
I20260926 09:27:25.193785 2023 raft_consensus.cc:2781] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:27:25.193825 2023 raft_consensus.cc:482] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:27:25.193858 2023 raft_consensus.cc:3037] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:25.194540 2023 raft_consensus.cc:504] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.194670 2023 leader_election.cc:288] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72 [CANDIDATE]: Term 1 election: Requested vote from peers 1844920c4227449fb5fb4248a7737a6d (127.0.98.2:37457), 43c7c9e34d1f4663a3af7ba1267dd38a (127.0.98.3:35357)
I20260926 09:27:25.194809 1799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d775f1a16be49c2811fb688dfb82c67" candidate_uuid: "716891f111124d66909210ac2e206a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1844920c4227449fb5fb4248a7737a6d"
I20260926 09:27:25.194834 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d775f1a16be49c2811fb688dfb82c67" candidate_uuid: "716891f111124d66909210ac2e206a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a"
I20260926 09:27:25.194900 1944 raft_consensus.cc:3037] T 5d775f1a16be49c2811fb688dfb82c67 P 43c7c9e34d1f4663a3af7ba1267dd38a [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:25.194909 1799 raft_consensus.cc:3037] T 5d775f1a16be49c2811fb688dfb82c67 P 1844920c4227449fb5fb4248a7737a6d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:25.195495 1799 raft_consensus.cc:2454] T 5d775f1a16be49c2811fb688dfb82c67 P 1844920c4227449fb5fb4248a7737a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716891f111124d66909210ac2e206a72 in term 1.
I20260926 09:27:25.195664 1944 raft_consensus.cc:2454] T 5d775f1a16be49c2811fb688dfb82c67 P 43c7c9e34d1f4663a3af7ba1267dd38a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716891f111124d66909210ac2e206a72 in term 1.
I20260926 09:27:25.195732 1595 leader_election.cc:302] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1844920c4227449fb5fb4248a7737a6d, 716891f111124d66909210ac2e206a72; no voters:
I20260926 09:27:25.195840 2023 raft_consensus.cc:2781] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:25.195899 2023 raft_consensus.cc:686] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72 [term 1 LEADER]: Becoming Leader. State: Replica: 716891f111124d66909210ac2e206a72, State: Running, Role: LEADER
I20260926 09:27:25.195964 2023 consensus_queue.cc:237] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.196578 1507 catalog_manager.cc:5495] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 716891f111124d66909210ac2e206a72 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "716891f111124d66909210ac2e206a72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } health_report { overall_health: UNKNOWN } } }
W20260926 09:27:25.199187 1991 tablet.cc:2369] T 5d775f1a16be49c2811fb688dfb82c67 P 43c7c9e34d1f4663a3af7ba1267dd38a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 09:27:25 dist-test-slave-dfzm krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:25.234241 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:25.226323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43609 (local address 127.0.0.1:59190)
0926 09:27:25.227025 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:25.227043 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:25.227140 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:25.227298 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:25.227299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:25.227655 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:25.227700 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:25.227703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.228372 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.228373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:25.228898 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:25.228899 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.229261 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.229262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:25.229319 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:25.229958 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:25.229963 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:25.231544 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:25.233118 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:25.233125 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:25.233128 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:25.233313 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:25.233519 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:25.233520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:25.233521 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:25.233623 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:25.233865 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:25.233873 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:25.233916 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 09:27:25.233991 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 09:27:25.234113 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":144,"threads_started":1}
I20260926 09:27:25.234241 1575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:25.226466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59190 (local address 127.0.98.62:43609)
0926 09:27:25.226582 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:25.226584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:25.227125 (+ 541us) server_negotiation.cc:407] Connection header received
0926 09:27:25.227215 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:25.227216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:25.227226 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:25.227295 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:25.227808 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.228291 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.228997 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.229177 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.229260 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:25.231624 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:25.231625 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:25.231626 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:25.231633 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:25.233037 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:25.233398 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:25.233399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:25.233400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:25.233453 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:25.233674 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:25.233675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:25.233675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:25.233785 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:25.233863 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:25.234032 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 09:27:25.234162 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":71}
I20260926 09:27:25.236434 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59190:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 09:27:25.236699 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:27:25.238265 1784 tablet_service.cc:1467] Processing CreateTablet for tablet ada80468704543778567179d07d99a30 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b017fd2b16b94207abff5e0ded48c9ff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:27:25.238312 1644 tablet_service.cc:1467] Processing CreateTablet for tablet ada80468704543778567179d07d99a30 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b017fd2b16b94207abff5e0ded48c9ff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:27:25.238301 1924 tablet_service.cc:1467] Processing CreateTablet for tablet ada80468704543778567179d07d99a30 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b017fd2b16b94207abff5e0ded48c9ff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:27:25.238435 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ada80468704543778567179d07d99a30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:25.238466 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ada80468704543778567179d07d99a30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:25.238461 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ada80468704543778567179d07d99a30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:25.239526 2016 tablet_bootstrap.cc:492] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a: Bootstrap starting.
I20260926 09:27:25.239879 2013 tablet_bootstrap.cc:492] T ada80468704543778567179d07d99a30 P 1844920c4227449fb5fb4248a7737a6d: Bootstrap starting.
I20260926 09:27:25.239960 2016 tablet_bootstrap.cc:654] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:25.240273 2013 tablet_bootstrap.cc:654] T ada80468704543778567179d07d99a30 P 1844920c4227449fb5fb4248a7737a6d: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:25.240609 2018 tablet_bootstrap.cc:492] T ada80468704543778567179d07d99a30 P 716891f111124d66909210ac2e206a72: Bootstrap starting.
I20260926 09:27:25.240671 2016 tablet_bootstrap.cc:492] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a: No bootstrap required, opened a new log
I20260926 09:27:25.240715 2016 ts_tablet_manager.cc:1397] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:25.240844 2013 tablet_bootstrap.cc:492] T ada80468704543778567179d07d99a30 P 1844920c4227449fb5fb4248a7737a6d: No bootstrap required, opened a new log
I20260926 09:27:25.240878 2013 ts_tablet_manager.cc:1397] T ada80468704543778567179d07d99a30 P 1844920c4227449fb5fb4248a7737a6d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:25.240859 2016 raft_consensus.cc:348] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.240916 2018 tablet_bootstrap.cc:654] T ada80468704543778567179d07d99a30 P 716891f111124d66909210ac2e206a72: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:25.240936 2016 raft_consensus.cc:374] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:25.240976 2016 raft_consensus.cc:729] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c7c9e34d1f4663a3af7ba1267dd38a, State: Initialized, Role: FOLLOWER
I20260926 09:27:25.241026 2016 consensus_queue.cc:260] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.241003 2013 raft_consensus.cc:348] T ada80468704543778567179d07d99a30 P 1844920c4227449fb5fb4248a7737a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.241086 2013 raft_consensus.cc:374] T ada80468704543778567179d07d99a30 P 1844920c4227449fb5fb4248a7737a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:25.241108 2013 raft_consensus.cc:729] T ada80468704543778567179d07d99a30 P 1844920c4227449fb5fb4248a7737a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1844920c4227449fb5fb4248a7737a6d, State: Initialized, Role: FOLLOWER
I20260926 09:27:25.241123 2016 ts_tablet_manager.cc:1428] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:25.241151 2013 consensus_queue.cc:260] T ada80468704543778567179d07d99a30 P 1844920c4227449fb5fb4248a7737a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.241233 2013 ts_tablet_manager.cc:1428] T ada80468704543778567179d07d99a30 P 1844920c4227449fb5fb4248a7737a6d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:25.241182 2020 tablet_replica.cc:439] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } } }
I20260926 09:27:25.241274 2020 tablet_replica.cc:442] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:27:25.241410 2015 tablet_replica.cc:439] T ada80468704543778567179d07d99a30 P 1844920c4227449fb5fb4248a7737a6d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } } }
I20260926 09:27:25.241513 2015 tablet_replica.cc:442] T ada80468704543778567179d07d99a30 P 1844920c4227449fb5fb4248a7737a6d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:27:25.241957 2018 tablet_bootstrap.cc:492] T ada80468704543778567179d07d99a30 P 716891f111124d66909210ac2e206a72: No bootstrap required, opened a new log
I20260926 09:27:25.242012 2018 ts_tablet_manager.cc:1397] T ada80468704543778567179d07d99a30 P 716891f111124d66909210ac2e206a72: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 09:27:25.242146 2018 raft_consensus.cc:348] T ada80468704543778567179d07d99a30 P 716891f111124d66909210ac2e206a72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.242226 2018 raft_consensus.cc:374] T ada80468704543778567179d07d99a30 P 716891f111124d66909210ac2e206a72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:25.242256 2018 raft_consensus.cc:729] T ada80468704543778567179d07d99a30 P 716891f111124d66909210ac2e206a72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716891f111124d66909210ac2e206a72, State: Initialized, Role: FOLLOWER
I20260926 09:27:25.242296 2018 consensus_queue.cc:260] T ada80468704543778567179d07d99a30 P 716891f111124d66909210ac2e206a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.242403 2018 ts_tablet_manager.cc:1428] T ada80468704543778567179d07d99a30 P 716891f111124d66909210ac2e206a72: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:25.242411 2023 tablet_replica.cc:439] T ada80468704543778567179d07d99a30 P 716891f111124d66909210ac2e206a72: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } } }
I20260926 09:27:25.242550 2023 tablet_replica.cc:442] T ada80468704543778567179d07d99a30 P 716891f111124d66909210ac2e206a72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:27:25.410641 2020 raft_consensus.cc:482] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:27:25.410737 2020 raft_consensus.cc:504] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.411164 2020 leader_election.cc:288] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 716891f111124d66909210ac2e206a72 (127.0.98.1:40879), 1844920c4227449fb5fb4248a7737a6d (127.0.98.2:37457)
I20260926 09:27:25.414263 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:25.411537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45187 (local address 127.0.98.1:40879)
0926 09:27:25.411589 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:25.411591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:25.411733 (+ 142us) server_negotiation.cc:407] Connection header received
0926 09:27:25.411763 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:25.411763 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:25.411773 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:25.411828 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:25.412014 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.412482 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.413437 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.413941 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.414002 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:25.414011 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:25.414179 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 09:27:25.414207 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:27:25.414312 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:25.411373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40879 (local address 127.0.98.3:45187)
0926 09:27:25.411626 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:25.411635 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:25.411685 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:25.411838 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:25.411839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:25.411840 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:25.411940 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:25.411941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.412560 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.412560 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:25.413345 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:25.413348 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.414013 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.414014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:25.414087 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:25.414092 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:27:25.414168 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 09:27:25.414207 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":17,"thread_start_us":162,"threads_started":1}
I20260926 09:27:25.414639 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:25.411373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37457 (local address 127.0.98.3:39427)
0926 09:27:25.411437 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:25.411452 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:25.411519 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:25.411732 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:25.411734 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:25.411736 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:25.411811 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:25.411814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.412516 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.412517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:25.413667 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:25.413669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.414329 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.414330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:25.414404 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:25.414407 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:27:25.414536 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 09:27:25.414579 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 09:27:25.414654 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:25.411539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39427 (local address 127.0.98.2:37457)
0926 09:27:25.411586 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:25.411589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:25.411597 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:27:25.411641 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:25.411642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:25.411653 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:25.411712 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:25.411881 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.412411 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.413783 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.414250 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.414315 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:25.414327 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:25.414547 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 09:27:25.414592 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:27:25.414925 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada80468704543778567179d07d99a30" candidate_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716891f111124d66909210ac2e206a72" is_pre_election: true
I20260926 09:27:25.414994 1799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada80468704543778567179d07d99a30" candidate_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1844920c4227449fb5fb4248a7737a6d" is_pre_election: true
I20260926 09:27:25.415086 1799 raft_consensus.cc:2454] T ada80468704543778567179d07d99a30 P 1844920c4227449fb5fb4248a7737a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43c7c9e34d1f4663a3af7ba1267dd38a in term 0.
I20260926 09:27:25.415103 1664 raft_consensus.cc:2454] T ada80468704543778567179d07d99a30 P 716891f111124d66909210ac2e206a72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43c7c9e34d1f4663a3af7ba1267dd38a in term 0.
I20260926 09:27:25.415313 1875 leader_election.cc:302] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1844920c4227449fb5fb4248a7737a6d, 43c7c9e34d1f4663a3af7ba1267dd38a; no voters:
I20260926 09:27:25.415445 2020 raft_consensus.cc:2781] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:27:25.415526 2020 raft_consensus.cc:482] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:27:25.415560 2020 raft_consensus.cc:3037] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:25.416252 2020 raft_consensus.cc:504] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.416437 2020 leader_election.cc:288] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a [CANDIDATE]: Term 1 election: Requested vote from peers 716891f111124d66909210ac2e206a72 (127.0.98.1:40879), 1844920c4227449fb5fb4248a7737a6d (127.0.98.2:37457)
I20260926 09:27:25.416574 1799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada80468704543778567179d07d99a30" candidate_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1844920c4227449fb5fb4248a7737a6d"
I20260926 09:27:25.416677 1799 raft_consensus.cc:3037] T ada80468704543778567179d07d99a30 P 1844920c4227449fb5fb4248a7737a6d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:25.416656 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada80468704543778567179d07d99a30" candidate_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716891f111124d66909210ac2e206a72"
I20260926 09:27:25.416862 1664 raft_consensus.cc:3037] T ada80468704543778567179d07d99a30 P 716891f111124d66909210ac2e206a72 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:25.417320 1799 raft_consensus.cc:2454] T ada80468704543778567179d07d99a30 P 1844920c4227449fb5fb4248a7737a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43c7c9e34d1f4663a3af7ba1267dd38a in term 1.
I20260926 09:27:25.417541 1875 leader_election.cc:302] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1844920c4227449fb5fb4248a7737a6d, 43c7c9e34d1f4663a3af7ba1267dd38a; no voters:
I20260926 09:27:25.417659 2020 raft_consensus.cc:2781] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:25.417704 1664 raft_consensus.cc:2454] T ada80468704543778567179d07d99a30 P 716891f111124d66909210ac2e206a72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43c7c9e34d1f4663a3af7ba1267dd38a in term 1.
I20260926 09:27:25.417819 2020 raft_consensus.cc:686] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a [term 1 LEADER]: Becoming Leader. State: Replica: 43c7c9e34d1f4663a3af7ba1267dd38a, State: Running, Role: LEADER
I20260926 09:27:25.417891 2020 consensus_queue.cc:237] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } }
I20260926 09:27:25.418256 2053 tablet_replica.cc:439] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a: TxnStatusTablet state changed. Reason: New leader 43c7c9e34d1f4663a3af7ba1267dd38a. Latest consensus state: current_term: 1 leader_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } } }
I20260926 09:27:25.418342 2053 tablet_replica.cc:442] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:27:25.418551 2055 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 09:27:25.418671 2055 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 09:27:25.418721 1506 catalog_manager.cc:5495] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43c7c9e34d1f4663a3af7ba1267dd38a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } health_report { overall_health: HEALTHY } } }
I20260926 09:27:25.424697 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:25.421753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56886 (local address 127.0.98.3:35357)
0926 09:27:25.421794 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:25.421796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:25.421830 (+ 34us) server_negotiation.cc:407] Connection header received
0926 09:27:25.421926 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:25.421927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:25.421940 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:25.422005 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:27:25.422141 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.422594 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.423736 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.424038 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.424113 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:25.424322 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:27:25.424447 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:27:25.424530 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:25.424604 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 09:27:25.424628 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:27:25.424741 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:25.421700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35357 (local address 127.0.0.1:56886)
0926 09:27:25.421775 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:25.421788 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:25.421851 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:25.422014 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:25.422015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:25.422017 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:27:25.422069 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:25.422071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.422672 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.422673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:25.423642 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:25.423646 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.424128 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.424129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:25.424200 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:25.424219 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:27:25.424531 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:27:25.424542 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 09:27:25.424602 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 09:27:25.424664 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 09:27:25.426169 1799 raft_consensus.cc:1261] T ada80468704543778567179d07d99a30 P 1844920c4227449fb5fb4248a7737a6d [term 1 FOLLOWER]: Refusing update from remote peer 43c7c9e34d1f4663a3af7ba1267dd38a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:25.426185 1664 raft_consensus.cc:1261] T ada80468704543778567179d07d99a30 P 716891f111124d66909210ac2e206a72 [term 1 FOLLOWER]: Refusing update from remote peer 43c7c9e34d1f4663a3af7ba1267dd38a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:25.426452 2053 consensus_queue.cc:1035] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:25.426605 2020 consensus_queue.cc:1035] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a [LEADER]: Connected to new peer: Peer: permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:25.427182 2015 tablet_replica.cc:439] T ada80468704543778567179d07d99a30 P 1844920c4227449fb5fb4248a7737a6d: TxnStatusTablet state changed. Reason: New leader 43c7c9e34d1f4663a3af7ba1267dd38a. Latest consensus state: current_term: 1 leader_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } } }
I20260926 09:27:25.427199 2023 tablet_replica.cc:439] T ada80468704543778567179d07d99a30 P 716891f111124d66909210ac2e206a72: TxnStatusTablet state changed. Reason: New leader 43c7c9e34d1f4663a3af7ba1267dd38a. Latest consensus state: current_term: 1 leader_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } } }
I20260926 09:27:25.427278 2015 tablet_replica.cc:442] T ada80468704543778567179d07d99a30 P 1844920c4227449fb5fb4248a7737a6d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:27:25.427289 2023 tablet_replica.cc:442] T ada80468704543778567179d07d99a30 P 716891f111124d66909210ac2e206a72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:27:25.427667 2020 tablet_replica.cc:439] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } } }
I20260926 09:27:25.427747 2020 tablet_replica.cc:442] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:27:25.428112 2015 tablet_replica.cc:439] T ada80468704543778567179d07d99a30 P 1844920c4227449fb5fb4248a7737a6d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } } }
I20260926 09:27:25.428200 2015 tablet_replica.cc:442] T ada80468704543778567179d07d99a30 P 1844920c4227449fb5fb4248a7737a6d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:27:25.428882 2053 tablet_replica.cc:439] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } } }
I20260926 09:27:25.428969 2053 tablet_replica.cc:442] T ada80468704543778567179d07d99a30 P 43c7c9e34d1f4663a3af7ba1267dd38a: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:27:25.429443 2023 tablet_replica.cc:439] T ada80468704543778567179d07d99a30 P 716891f111124d66909210ac2e206a72: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716891f111124d66909210ac2e206a72" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40879 } } peers { permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 } } peers { permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 } } }
I20260926 09:27:25.429520 2023 tablet_replica.cc:442] T ada80468704543778567179d07d99a30 P 716891f111124d66909210ac2e206a72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:27:25.434137 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:25.431212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39279 (local address 127.0.98.3:35357)
0926 09:27:25.431254 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:25.431257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:25.431400 (+ 143us) server_negotiation.cc:407] Connection header received
0926 09:27:25.431454 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:25.431455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:25.431464 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:25.431505 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:27:25.431633 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.432263 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.433466 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.433727 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.433783 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:25.433958 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:27:25.433988 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:27:25.434022 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:25.434074 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 09:27:25.434090 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:27:25.434203 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:25.431137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35357 (local address 127.0.98.1:39279)
0926 09:27:25.431323 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:25.431335 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:25.431398 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:25.431517 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:25.431518 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:25.431520 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:27:25.431570 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:25.431572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.432364 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.432366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:25.433386 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:25.433390 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.433797 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.433798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:25.433858 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:25.433874 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:27:25.434023 (+ 149us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:27:25.434024 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 09:27:25.434074 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 09:27:25.434120 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":108,"threads_started":1}
I20260926 09:27:25.436684 1944 raft_consensus.cc:1261] T 5d775f1a16be49c2811fb688dfb82c67 P 43c7c9e34d1f4663a3af7ba1267dd38a [term 1 FOLLOWER]: Refusing update from remote peer 716891f111124d66909210ac2e206a72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:25.436821 1799 raft_consensus.cc:1261] T 5d775f1a16be49c2811fb688dfb82c67 P 1844920c4227449fb5fb4248a7737a6d [term 1 FOLLOWER]: Refusing update from remote peer 716891f111124d66909210ac2e206a72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:25.436915 2023 consensus_queue.cc:1035] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43c7c9e34d1f4663a3af7ba1267dd38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:25.437001 2021 consensus_queue.cc:1035] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1844920c4227449fb5fb4248a7737a6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:25.445350 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:25.442662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58339 (local address 127.0.98.1:40879)
0926 09:27:25.442698 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:25.442700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:25.442710 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:27:25.442743 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:25.442744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:25.442756 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:25.442806 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:27:25.442909 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.443653 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.444647 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.444848 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.444924 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:25.445103 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:27:25.445138 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:27:25.445188 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:25.445251 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 09:27:25.445275 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:27:25.445361 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:25.442564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40879 (local address 127.0.98.3:58339)
0926 09:27:25.442625 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:25.442648 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:25.442690 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:25.442809 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:25.442810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:25.442810 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:27:25.442865 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:25.442866 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.443751 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.443752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:25.444559 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:25.444561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.444920 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.444921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:25.444980 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:25.445010 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:27:25.445198 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:27:25.445199 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 09:27:25.445239 (+ 40us) client_negotiation.cc:240] Negotiation successful
0926 09:27:25.445298 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 09:27:25.454190 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:25.454305 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:25.455034 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81904029a2ff42e988ef366158b0bb62: Sending DeleteTablet for 3 replicas of tablet 5d775f1a16be49c2811fb688dfb82c67
I20260926 09:27:25.455307 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d775f1a16be49c2811fb688dfb82c67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54358
I20260926 09:27:25.455350 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d775f1a16be49c2811fb688dfb82c67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56872
I20260926 09:27:25.455425 2036 tablet_replica.cc:330] T 5d775f1a16be49c2811fb688dfb82c67 P 1844920c4227449fb5fb4248a7737a6d: stopping tablet replica
I20260926 09:27:25.455457 2038 tablet_replica.cc:330] T 5d775f1a16be49c2811fb688dfb82c67 P 43c7c9e34d1f4663a3af7ba1267dd38a: stopping tablet replica
I20260926 09:27:25.455466 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d775f1a16be49c2811fb688dfb82c67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52276
I20260926 09:27:25.455488 2036 raft_consensus.cc:2229] T 5d775f1a16be49c2811fb688dfb82c67 P 1844920c4227449fb5fb4248a7737a6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:25.455497 2038 raft_consensus.cc:2229] T 5d775f1a16be49c2811fb688dfb82c67 P 43c7c9e34d1f4663a3af7ba1267dd38a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:25.455528 2038 raft_consensus.cc:2258] T 5d775f1a16be49c2811fb688dfb82c67 P 43c7c9e34d1f4663a3af7ba1267dd38a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:25.455528 2036 raft_consensus.cc:2258] T 5d775f1a16be49c2811fb688dfb82c67 P 1844920c4227449fb5fb4248a7737a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:25.455561 2037 tablet_replica.cc:330] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72: stopping tablet replica
I20260926 09:27:25.455622 2037 raft_consensus.cc:2229] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:27:25.455684 2037 raft_consensus.cc:2258] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:25.455866 2038 ts_tablet_manager.cc:1905] T 5d775f1a16be49c2811fb688dfb82c67 P 43c7c9e34d1f4663a3af7ba1267dd38a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:25.456060 2037 ts_tablet_manager.cc:1905] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:25.457755 2038 ts_tablet_manager.cc:1918] T 5d775f1a16be49c2811fb688dfb82c67 P 43c7c9e34d1f4663a3af7ba1267dd38a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:27:25.457810 2038 log.cc:1196] T 5d775f1a16be49c2811fb688dfb82c67 P 43c7c9e34d1f4663a3af7ba1267dd38a: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/5d775f1a16be49c2811fb688dfb82c67
I20260926 09:27:25.458065 2038 ts_tablet_manager.cc:1939] T 5d775f1a16be49c2811fb688dfb82c67 P 43c7c9e34d1f4663a3af7ba1267dd38a: Deleting consensus metadata
I20260926 09:27:25.458199 2037 ts_tablet_manager.cc:1918] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:27:25.458251 2037 log.cc:1196] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-0/wal/wals/5d775f1a16be49c2811fb688dfb82c67
I20260926 09:27:25.458371 1491 catalog_manager.cc:4841] TS 43c7c9e34d1f4663a3af7ba1267dd38a (127.0.98.3:35357): tablet 5d775f1a16be49c2811fb688dfb82c67 (table test-table [id=958c7dcedb9a485e9be2b96180b62e2a]) successfully deleted
I20260926 09:27:25.458455 2037 ts_tablet_manager.cc:1939] T 5d775f1a16be49c2811fb688dfb82c67 P 716891f111124d66909210ac2e206a72: Deleting consensus metadata
I20260926 09:27:25.458606 2036 ts_tablet_manager.cc:1905] T 5d775f1a16be49c2811fb688dfb82c67 P 1844920c4227449fb5fb4248a7737a6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:25.460093 2036 ts_tablet_manager.cc:1918] T 5d775f1a16be49c2811fb688dfb82c67 P 1844920c4227449fb5fb4248a7737a6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:27:25.460166 2036 log.cc:1196] T 5d775f1a16be49c2811fb688dfb82c67 P 1844920c4227449fb5fb4248a7737a6d: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414839069479-392-0/minicluster-data/ts-1/wal/wals/5d775f1a16be49c2811fb688dfb82c67
I20260926 09:27:25.460368 2036 ts_tablet_manager.cc:1939] T 5d775f1a16be49c2811fb688dfb82c67 P 1844920c4227449fb5fb4248a7737a6d: Deleting consensus metadata
I20260926 09:27:25.460731 1493 catalog_manager.cc:4841] TS 716891f111124d66909210ac2e206a72 (127.0.98.1:40879): tablet 5d775f1a16be49c2811fb688dfb82c67 (table test-table [id=958c7dcedb9a485e9be2b96180b62e2a]) successfully deleted
I20260926 09:27:25.460817 1491 catalog_manager.cc:4841] TS 1844920c4227449fb5fb4248a7737a6d (127.0.98.2:37457): tablet 5d775f1a16be49c2811fb688dfb82c67 (table test-table [id=958c7dcedb9a485e9be2b96180b62e2a]) successfully deleted
Sep 26 09:27:25 dist-test-slave-dfzm krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414844, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 09:27:25.466104 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:25.456599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52292 (local address 127.0.98.1:40879)
0926 09:27:25.456639 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:25.456641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:25.457328 (+ 687us) server_negotiation.cc:407] Connection header received
0926 09:27:25.457445 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:25.457445 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:25.457459 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:25.457490 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:25.458315 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.458992 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.459684 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.459892 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.459981 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:25.463180 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:25.463189 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:25.463190 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:25.463197 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:25.464848 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:25.465211 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:25.465217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:25.465217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:25.465290 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:25.465500 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:25.465501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:25.465501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:25.465708 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:25.465860 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:25.465910 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 09:27:25.465991 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 09:27:25.466346 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52292
I20260926 09:27:25.473477 1575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:25.467300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59192 (local address 127.0.98.62:43609)
0926 09:27:25.467351 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:25.467353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:25.467382 (+ 29us) server_negotiation.cc:407] Connection header received
0926 09:27:25.467451 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:25.467452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:25.467462 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:25.467511 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:25.467980 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.468737 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.469401 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:25.469731 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:25.469823 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:25.470990 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:25.470992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:25.470992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:25.471000 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:25.472498 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:25.472801 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:25.472802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:25.472803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:25.472863 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:25.473030 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:25.473031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:25.473032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:25.473155 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:25.473249 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:25.473312 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 09:27:25.473382 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260926 09:27:25.473693 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59192
I20260926 09:27:25.474331 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 1579
I20260926 09:27:25.481925 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 1719
I20260926 09:27:25.489748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 1859
I20260926 09:27:25.494761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 1475
2026-09-26T09:27:25Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2369 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:27:25 dist-test-slave-dfzm krb5kdc[2074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:27:25 dist-test-slave-dfzm krb5kdc[2074](info): set up 2 sockets
Sep 26 09:27:25 dist-test-slave-dfzm krb5kdc[2074](info): commencing operation
krb5kdc: starting...
W20260926 09:27:26.537284 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:27:26 dist-test-slave-dfzm krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:27:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:27:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:27:26.657445 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44869
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:26.738724 2087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:26.738935 2087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:26.738958 2087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:26.740052 2087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:27:26.740089 2087 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 09:27:26.740103 2087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:26.740115 2087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:26.740128 2087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:27:26.740139 2087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:27:26.741526 2087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44869
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:26.741819 2087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:26.742072 2087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:26.747030 2093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:26.747171 2095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:26.747303 2087 server_base.cc:1034] running on GCE node
W20260926 09:27:26.747030 2092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:26.747920 2087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:26.748231 2087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:26.749404 2087 hybrid_clock.cc:648] HybridClock initialized: now 1790414846749398 us; error 43 us; skew 500 ppm
Sep 26 09:27:26 dist-test-slave-dfzm krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:26.751968 2087 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:27:26.752429 2087 webserver.cc:460] Webserver started at http://127.0.98.62:42833/ using document root <none> and password file <none>
I20260926 09:27:26.752629 2087 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:26.752669 2087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:26.752803 2087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:26.753907 2087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "1c8cf53869c446919b7d2de4560aeb2f"
format_stamp: "Formatted at 2026-09-26 09:27:26 on dist-test-slave-dfzm"
server_key: "dd917a617229188e02f8b8b038518a6f"
server_key_iv: "281c01712974767abed9120a71e16bdd"
server_key_version: "encryptionkey@0"
I20260926 09:27:26.754202 2087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "1c8cf53869c446919b7d2de4560aeb2f"
format_stamp: "Formatted at 2026-09-26 09:27:26 on dist-test-slave-dfzm"
server_key: "dd917a617229188e02f8b8b038518a6f"
server_key_iv: "281c01712974767abed9120a71e16bdd"
server_key_version: "encryptionkey@0"
I20260926 09:27:26.755465 2087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:26.756186 2102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:26.756392 2087 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:26.756476 2087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "1c8cf53869c446919b7d2de4560aeb2f"
format_stamp: "Formatted at 2026-09-26 09:27:26 on dist-test-slave-dfzm"
server_key: "dd917a617229188e02f8b8b038518a6f"
server_key_iv: "281c01712974767abed9120a71e16bdd"
server_key_version: "encryptionkey@0"
I20260926 09:27:26.756536 2087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:26.765093 2087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:26.766957 2087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:26.767069 2087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:26.771440 2087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44869
I20260926 09:27:26.771519 2164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44869 every 8 connection(s)
I20260926 09:27:26.771919 2087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:27:26.772617 2165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:26.774984 2165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f: Bootstrap starting.
I20260926 09:27:26.775589 2165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:26.775876 2165 log.cc:824] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:26.776561 2165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f: No bootstrap required, opened a new log
I20260926 09:27:26.778715 2165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8cf53869c446919b7d2de4560aeb2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } }
I20260926 09:27:26.778887 2165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:26.778932 2165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c8cf53869c446919b7d2de4560aeb2f, State: Initialized, Role: FOLLOWER
I20260926 09:27:26.779110 2165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8cf53869c446919b7d2de4560aeb2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } }
I20260926 09:27:26.779196 2165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:27:26.779247 2165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:27:26.779300 2165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:26.779852 2165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8cf53869c446919b7d2de4560aeb2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } }
I20260926 09:27:26.779976 2165 leader_election.cc:302] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c8cf53869c446919b7d2de4560aeb2f; no voters:
I20260926 09:27:26.780164 2165 leader_election.cc:288] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:27:26.780210 2168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:26.780375 2168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f [term 1 LEADER]: Becoming Leader. State: Replica: 1c8cf53869c446919b7d2de4560aeb2f, State: Running, Role: LEADER
I20260926 09:27:26.780447 2165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:27:26.780459 2168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8cf53869c446919b7d2de4560aeb2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } }
I20260926 09:27:26.780906 2170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c8cf53869c446919b7d2de4560aeb2f. Latest consensus state: current_term: 1 leader_uuid: "1c8cf53869c446919b7d2de4560aeb2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8cf53869c446919b7d2de4560aeb2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } }
I20260926 09:27:26.781003 2170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:26.780896 2169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c8cf53869c446919b7d2de4560aeb2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8cf53869c446919b7d2de4560aeb2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44869 } } }
I20260926 09:27:26.781157 2169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:26.781361 2177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:27:26.781919 2177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:27:26.781993 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 2087
I20260926 09:27:26.782081 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:27:26.782253 392 external_mini_cluster.cc:1428] Setting key f7bb504b580332a428d2929a127ba045
I20260926 09:27:26.783634 2177 catalog_manager.cc:1347] Generated new cluster ID: 5b5f8b0254564409bbde05d0226d0f55
I20260926 09:27:26.783689 2177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:27:26 dist-test-slave-dfzm krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:26.793741 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:26.782703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52358 (local address 127.0.98.62:44869)
0926 09:27:26.783087 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:26.783098 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:26.783136 (+ 38us) server_negotiation.cc:407] Connection header received
0926 09:27:26.783256 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:26.783263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:26.783334 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:26.783474 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:26.784203 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:26.785635 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:26.786655 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:26.787247 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:26.787617 (+ 370us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:26.790123 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:26.790127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:26.790127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:26.790138 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:26.792079 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:26.792487 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:26.792489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:26.792490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:26.792590 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:26.792803 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:26.792804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:26.792804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:26.793106 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:26.793229 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:26.793357 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 09:27:26.793482 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":80,"threads_started":1}
I20260926 09:27:26.795672 2177 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:27:26.796144 2177 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:27:26.805130 2177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f: Generated new TSK 0
I20260926 09:27:26.805346 2177 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:27:26.836920 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44869
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:26.918321 2191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:26.918531 2191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:26.918556 2191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:26.919705 2191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:26.919749 2191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:26.919770 2191 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 09:27:26.919809 2191 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:27:26.919844 2191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:27:26.921337 2191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:26.921572 2191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:26.921839 2191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:26.926915 2199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:26.926870 2196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:26.926877 2197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:26.927078 2191 server_base.cc:1034] running on GCE node
I20260926 09:27:26.927304 2191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:26.927508 2191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:26.928646 2191 hybrid_clock.cc:648] HybridClock initialized: now 1790414846928636 us; error 31 us; skew 500 ppm
Sep 26 09:27:26 dist-test-slave-dfzm krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:26.931264 2191 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:27:26.931679 2191 webserver.cc:460] Webserver started at http://127.0.98.1:36361/ using document root <none> and password file <none>
I20260926 09:27:26.931900 2191 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:26.931943 2191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:26.932093 2191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:26.933158 2191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "519583d12f9c4e7bac953975b7c7c6ec"
format_stamp: "Formatted at 2026-09-26 09:27:26 on dist-test-slave-dfzm"
server_key: "83b5e3b668f5adbaf7a4c3e08fe16d42"
server_key_iv: "11a72cdffc76a6722d4741a0c904f2c5"
server_key_version: "encryptionkey@0"
I20260926 09:27:26.933525 2191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "519583d12f9c4e7bac953975b7c7c6ec"
format_stamp: "Formatted at 2026-09-26 09:27:26 on dist-test-slave-dfzm"
server_key: "83b5e3b668f5adbaf7a4c3e08fe16d42"
server_key_iv: "11a72cdffc76a6722d4741a0c904f2c5"
server_key_version: "encryptionkey@0"
I20260926 09:27:26.934849 2191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:27:26.935539 2206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:26.935704 2191 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:27:26.935823 2191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "519583d12f9c4e7bac953975b7c7c6ec"
format_stamp: "Formatted at 2026-09-26 09:27:26 on dist-test-slave-dfzm"
server_key: "83b5e3b668f5adbaf7a4c3e08fe16d42"
server_key_iv: "11a72cdffc76a6722d4741a0c904f2c5"
server_key_version: "encryptionkey@0"
I20260926 09:27:26.935901 2191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:26.964278 2191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:26.966187 2191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:26.966367 2191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:26.966928 2191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:26.966991 2191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:26.967036 2191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:26.967082 2191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 09:27:26 dist-test-slave-dfzm krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:26.974251 2191 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34803
I20260926 09:27:26.974336 2321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34803 every 8 connection(s)
I20260926 09:27:26.974624 2191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:27:26.977766 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:26.967494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49267 (local address 127.0.98.62:44869)
0926 09:27:26.967543 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:26.967545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:26.967804 (+ 259us) server_negotiation.cc:407] Connection header received
0926 09:27:26.968412 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:26.968414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:26.968432 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:26.968508 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:26.969425 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:26.970288 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:26.971330 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:26.971637 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:26.971713 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:26.974713 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:26.974715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:26.974715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:26.974725 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:26.976303 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:26.976758 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:26.976760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:26.976760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:26.976817 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:26.977081 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:26.977081 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:26.977082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:26.977212 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:26.977282 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:26.977569 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 09:27:26.977675 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:27:26.977963 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:26.967494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44869 (local address 127.0.98.1:49267)
0926 09:27:26.967679 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:26.967714 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:26.968252 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:26.968601 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:26.968606 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:26.968992 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:26.969312 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:26.969318 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:26.970398 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:26.970399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:26.971209 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:26.971210 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:26.971736 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:26.971737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:26.971871 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:26.972458 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:26.972470 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:26.974557 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:26.976410 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:26.976417 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:26.976419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:26.976665 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:26.976899 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:26.976900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:26.976900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:26.977007 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:26.977286 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:26.977293 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:26.977428 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 09:27:26.977536 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 09:27:26.977730 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":95,"threads_started":1}
I20260926 09:27:26.978461 2322 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44869
I20260926 09:27:26.978544 2322 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:26.978734 2322 heartbeater.cc:511] Master 127.0.98.62:44869 requested a full tablet report, sending...
I20260926 09:27:26.979319 2118 ts_manager.cc:194] Registered new tserver with Master: 519583d12f9c4e7bac953975b7c7c6ec (127.0.98.1:34803)
I20260926 09:27:26.980232 2118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49267
I20260926 09:27:26.981801 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 2191
I20260926 09:27:26.981911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:27:26.982107 392 external_mini_cluster.cc:1428] Setting key a99fc99c42df8790dd8ee9caa5cb4768
I20260926 09:27:26.985685 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:26.979532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46851 (local address 127.0.98.62:44869)
0926 09:27:26.979578 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:26.979580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:26.979715 (+ 135us) server_negotiation.cc:407] Connection header received
0926 09:27:26.979770 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:26.979771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:26.979779 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:26.979825 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:26.980133 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:26.980779 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:26.981551 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:26.981814 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:26.981868 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:26.982969 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:26.982971 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:26.982971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:26.982978 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:26.984523 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:26.984951 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:26.984952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:26.984953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:26.985000 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:26.985188 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:26.985188 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:26.985188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:26.985292 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:26.985341 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:26.985502 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 09:27:26.985564 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:27:26.985773 2328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:26.979484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44869 (local address 127.0.98.1:46851)
0926 09:27:26.979639 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:26.979650 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:26.979714 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:26.979830 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:26.979830 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:26.980018 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:26.980075 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:26.980076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:26.980859 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:26.980860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:26.981484 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:26.981486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:26.981926 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:26.981928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:26.982023 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:26.982518 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:26.982522 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:26.982888 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:26.984604 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:26.984605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:26.984605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:26.984873 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:26.985061 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:26.985062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:26.985062 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:26.985128 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:26.985355 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:26.985356 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:26.985405 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 09:27:26.985477 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 09:27:26.985613 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:27:27.009501 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44869
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:27.090744 2331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:27.090961 2331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:27.090988 2331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:27.092155 2331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:27.092199 2331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:27.092221 2331 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 09:27:27.092232 2331 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:27:27.092299 2331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:27:27.093818 2331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:27.094061 2331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:27.094302 2331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:27.099403 2337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:27.099416 2336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:27.099689 2331 server_base.cc:1034] running on GCE node
W20260926 09:27:27.099839 2339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:27.100060 2331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:27.100288 2331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:27.101435 2331 hybrid_clock.cc:648] HybridClock initialized: now 1790414847101414 us; error 38 us; skew 500 ppm
Sep 26 09:27:27 dist-test-slave-dfzm krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:27.103936 2331 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:27:27.104384 2331 webserver.cc:460] Webserver started at http://127.0.98.2:40057/ using document root <none> and password file <none>
I20260926 09:27:27.104561 2331 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:27.104599 2331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:27.104684 2331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:27.105718 2331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "4712dcc9dcc54da19007ae2c1832554c"
format_stamp: "Formatted at 2026-09-26 09:27:27 on dist-test-slave-dfzm"
server_key: "f1208b92151441fe7bc94f65370cd144"
server_key_iv: "2884eb7db26fba9e9ef5f7f8afac8de5"
server_key_version: "encryptionkey@0"
I20260926 09:27:27.106015 2331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4712dcc9dcc54da19007ae2c1832554c"
format_stamp: "Formatted at 2026-09-26 09:27:27 on dist-test-slave-dfzm"
server_key: "f1208b92151441fe7bc94f65370cd144"
server_key_iv: "2884eb7db26fba9e9ef5f7f8afac8de5"
server_key_version: "encryptionkey@0"
I20260926 09:27:27.107231 2331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:27:27.107884 2346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:27.108064 2331 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:27:27.108151 2331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "4712dcc9dcc54da19007ae2c1832554c"
format_stamp: "Formatted at 2026-09-26 09:27:27 on dist-test-slave-dfzm"
server_key: "f1208b92151441fe7bc94f65370cd144"
server_key_iv: "2884eb7db26fba9e9ef5f7f8afac8de5"
server_key_version: "encryptionkey@0"
I20260926 09:27:27.108240 2331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:27.121160 2331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:27.123574 2331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:27.123730 2331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:27.124297 2331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:27.124337 2331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:27.124359 2331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:27.124372 2331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 09:27:27 dist-test-slave-dfzm krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:27.131698 2331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36885
I20260926 09:27:27.131793 2461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36885 every 8 connection(s)
I20260926 09:27:27.132117 2331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:27:27.134384 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.124703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39759 (local address 127.0.98.62:44869)
0926 09:27:27.124780 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:27.124782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:27.125029 (+ 247us) server_negotiation.cc:407] Connection header received
0926 09:27:27.125569 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:27.125570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:27.125581 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:27.125669 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:27.126254 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.126784 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.128037 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.128241 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.128324 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.131507 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:27.131508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:27.131509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:27.131518 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:27.133104 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:27.133452 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:27.133453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:27.133453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:27.133502 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:27.133771 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:27.133772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:27.133772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:27.133884 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:27.133937 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:27.134163 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 09:27:27.134282 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 09:27:27.134400 2355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.124696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44869 (local address 127.0.98.2:39759)
0926 09:27:27.124909 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:27.124948 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:27.125458 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:27.125742 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:27.125745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:27.125977 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:27.126154 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:27.126157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.126882 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.126883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:27.127931 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:27.127934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.128339 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.128340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:27.128460 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.128974 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:27.128985 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:27.131384 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:27.133175 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:27.133179 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:27.133180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:27.133385 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:27.133544 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:27.133544 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:27.133545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:27.133693 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:27.133944 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:27.133948 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:27.134057 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 09:27:27.134137 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 09:27:27.134253 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":149,"threads_started":1}
I20260926 09:27:27.134382 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 2331
I20260926 09:27:27.134482 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:27:27.134688 392 external_mini_cluster.cc:1428] Setting key db0aa1b83f3e6bd451e3654f1d26fb6e
I20260926 09:27:27.134965 2462 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44869
I20260926 09:27:27.135247 2462 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:27.136683 2462 heartbeater.cc:511] Master 127.0.98.62:44869 requested a full tablet report, sending...
I20260926 09:27:27.137025 2119 ts_manager.cc:194] Registered new tserver with Master: 4712dcc9dcc54da19007ae2c1832554c (127.0.98.2:36885)
I20260926 09:27:27.137545 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39759
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 09:27:27.142976 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.135876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41113 (local address 127.0.98.62:44869)
0926 09:27:27.135911 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:27.135913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:27.136961 (+ 1048us) server_negotiation.cc:407] Connection header received
0926 09:27:27.137011 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:27.137012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:27.137023 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:27.137090 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:27.137438 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.138125 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.139006 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.139222 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.139304 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.140336 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:27.140337 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:27.140338 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:27.140344 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:27.141750 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:27.142143 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:27.142144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:27.142144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:27.142199 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:27.142415 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:27.142416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:27.142416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:27.142567 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:27.142646 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:27.142802 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 09:27:27.142879 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 09:27:27.142992 2468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.135827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44869 (local address 127.0.98.2:41113)
0926 09:27:27.136889 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:27.136899 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:27.136957 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:27.137105 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:27.137106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:27.137296 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:27.137359 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:27.137361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.138218 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.138218 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:27.138914 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:27.138916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.139304 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.139305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:27.139380 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.139850 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:27.139855 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:27.140134 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:27.141877 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:27.141878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:27.141879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:27.142055 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:27.142278 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:27.142279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:27.142279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:27.142351 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:27.142637 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:27.142638 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:27.142685 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 09:27:27.142774 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 09:27:27.142880 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1040,"mutex_wait_us":9,"thread_start_us":73,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:27:27.160528 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44869
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:27.240872 2471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:27.241115 2471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:27.241154 2471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:27.242357 2471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:27.242424 2471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:27.242457 2471 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 09:27:27.242488 2471 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:27:27.242523 2471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:27:27.243961 2471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:27.244235 2471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:27.244467 2471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:27.249478 2477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:27.249470 2479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:27.249470 2476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:27.249573 2471 server_base.cc:1034] running on GCE node
I20260926 09:27:27.249876 2471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:27.250140 2471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:27.251328 2471 hybrid_clock.cc:648] HybridClock initialized: now 1790414847251305 us; error 38 us; skew 500 ppm
Sep 26 09:27:27 dist-test-slave-dfzm krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:27.253929 2471 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:27:27.254423 2471 webserver.cc:460] Webserver started at http://127.0.98.3:44101/ using document root <none> and password file <none>
I20260926 09:27:27.254657 2471 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:27.254724 2471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:27.254875 2471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:27.255923 2471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "eb0bc8efbdfd4537a1ae503077a60c4d"
format_stamp: "Formatted at 2026-09-26 09:27:27 on dist-test-slave-dfzm"
server_key: "fe92c7dcda619095ff97d1219b7c2cbe"
server_key_iv: "b42b8578829cdb130c334e1a0aeb24b9"
server_key_version: "encryptionkey@0"
I20260926 09:27:27.256294 2471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "eb0bc8efbdfd4537a1ae503077a60c4d"
format_stamp: "Formatted at 2026-09-26 09:27:27 on dist-test-slave-dfzm"
server_key: "fe92c7dcda619095ff97d1219b7c2cbe"
server_key_iv: "b42b8578829cdb130c334e1a0aeb24b9"
server_key_version: "encryptionkey@0"
I20260926 09:27:27.257673 2471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:27:27.258423 2486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:27.258603 2471 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:27.258706 2471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "eb0bc8efbdfd4537a1ae503077a60c4d"
format_stamp: "Formatted at 2026-09-26 09:27:27 on dist-test-slave-dfzm"
server_key: "fe92c7dcda619095ff97d1219b7c2cbe"
server_key_iv: "b42b8578829cdb130c334e1a0aeb24b9"
server_key_version: "encryptionkey@0"
I20260926 09:27:27.258777 2471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:27.272490 2471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:27.274902 2471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:27.275069 2471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:27.275578 2471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:27.275645 2471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:27.275694 2471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:27.275719 2471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 09:27:27 dist-test-slave-dfzm krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:27.283106 2471 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34015
I20260926 09:27:27.283478 2471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:27:27.283512 2601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34015 every 8 connection(s)
I20260926 09:27:27.285436 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 2471
I20260926 09:27:27.285552 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:27:27.285666 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.276120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41249 (local address 127.0.98.62:44869)
0926 09:27:27.276191 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:27.276194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:27.276393 (+ 199us) server_negotiation.cc:407] Connection header received
0926 09:27:27.276927 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:27.276928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:27.276941 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:27.277004 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:27.277970 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.278478 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.279643 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.279902 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.279993 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.282652 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:27.282654 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:27.282655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:27.282663 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:27.284284 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:27.284664 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:27.284665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:27.284666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:27.284719 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:27.284961 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:27.284963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:27.284963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:27.285084 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:27.285182 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:27.285438 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 09:27:27.285530 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 09:27:27.285705 2495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.276105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44869 (local address 127.0.98.3:41249)
0926 09:27:27.276277 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:27.276320 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:27.276821 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:27.277085 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:27.277088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:27.277485 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:27.277860 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:27.277865 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.278578 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.278579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:27.279540 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:27.279543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.280009 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.280010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:27.280190 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.280848 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:27.280869 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:27.282552 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:27.284358 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:27.284363 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:27.284364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:27.284573 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:27.284777 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:27.284777 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:27.284777 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:27.284879 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:27.285152 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:27.285155 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:27.285281 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 09:27:27.285367 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 09:27:27.285486 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":108,"threads_started":1}
I20260926 09:27:27.285796 392 external_mini_cluster.cc:1428] Setting key d4b8edf6f04bbabfd5bdfb0bb1560694
I20260926 09:27:27.287222 2602 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44869
I20260926 09:27:27.287310 2602 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:27.287482 2602 heartbeater.cc:511] Master 127.0.98.62:44869 requested a full tablet report, sending...
I20260926 09:27:27.287796 2118 ts_manager.cc:194] Registered new tserver with Master: eb0bc8efbdfd4537a1ae503077a60c4d (127.0.98.3:34015)
I20260926 09:27:27.288276 2118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41249
I20260926 09:27:27.292845 2608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.287327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44869 (local address 127.0.98.3:47225)
0926 09:27:27.287533 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:27.287543 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:27.287619 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:27.287797 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:27.287798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:27.287994 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:27.288059 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:27.288061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.288831 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.288832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:27.289454 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:27.289456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.289835 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.289835 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:27.289889 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.290257 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:27.290261 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:27.290570 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:27.292099 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:27.292099 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:27.292100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:27.292268 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:27.292421 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:27.292422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:27.292422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:27.292464 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:27.292625 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:27.292625 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:27.292659 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 09:27:27.292706 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 09:27:27.292776 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":141,"threads_started":1}
I20260926 09:27:27.292846 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.287390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47225 (local address 127.0.98.62:44869)
0926 09:27:27.287436 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:27.287438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:27.287607 (+ 169us) server_negotiation.cc:407] Connection header received
0926 09:27:27.287700 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:27.287701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:27.287711 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:27.287758 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:27.288133 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.288737 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.289523 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.289756 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.289822 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.290645 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:27.290646 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:27.290646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:27.290652 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:27.292020 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:27.292333 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:27.292334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:27.292334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:27.292381 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:27.292497 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:27.292497 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:27.292497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:27.292578 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:27.292621 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:27.292719 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 09:27:27.292770 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 09:27:27.297428 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 09:27:27 dist-test-slave-dfzm krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 09:27:27 dist-test-slave-dfzm krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:27.320363 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.313356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52370 (local address 127.0.98.62:44869)
0926 09:27:27.313418 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:27.313420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:27.313429 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:27:27.313496 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:27.313497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:27.313507 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:27.313559 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:27.314153 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.314632 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.315357 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.315583 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.315675 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.317975 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:27.317977 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:27.317977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:27.317984 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:27.319292 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:27.319611 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:27.319612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:27.319612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:27.319703 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:27.319884 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:27.319885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:27.319885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:27.319988 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:27.320063 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:27.320169 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 09:27:27.320232 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:27:27.322073 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:27:27.323679 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:27:27.330442 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.326170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36885 (local address 127.0.0.1:50134)
0926 09:27:27.326466 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:27.326509 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:27.326584 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:27.326903 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:27.326904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:27.326906 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:27.326986 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:27.326988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.328424 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.328425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:27.329368 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:27.329371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.330119 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.330120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:27.330217 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.330232 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 09:27:27.330307 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 09:27:27.330355 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":19,"thread_start_us":52,"threads_started":1}
I20260926 09:27:27.330457 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.326275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50134 (local address 127.0.98.2:36885)
0926 09:27:27.326548 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:27.326554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:27.326566 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:27:27.326730 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:27.326733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:27.326748 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:27.326822 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:27.327275 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.328358 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.329457 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.330032 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.330121 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.330206 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:27.330321 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 09:27:27.330355 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":88,"threads_started":1}
I20260926 09:27:27.330708 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.326170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34803 (local address 127.0.0.1:53432)
0926 09:27:27.326521 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:27.326531 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:27.326585 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:27.326899 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:27.326903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:27.326905 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:27.326984 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:27.326986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.328357 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.328358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:27.329436 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:27.329437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.330437 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.330438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:27.330510 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.330514 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:27:27.330575 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 09:27:27.330624 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":8,"thread_start_us":93,"threads_started":1}
I20260926 09:27:27.330739 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.326286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53432 (local address 127.0.98.1:34803)
0926 09:27:27.326673 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:27.326679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:27.326689 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:27:27.326730 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:27.326732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:27.326748 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:27.326889 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:27.327104 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.328254 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.329536 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.330269 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.330343 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.330397 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:27.330591 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 09:27:27.330636 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":80,"threads_started":1}
I20260926 09:27:27.331257 2396 tablet_service.cc:1467] Processing CreateTablet for tablet a04dd133580543b186f9346867107b1c (DEFAULT_TABLE table=test-table [id=3b43c41458bc4e8690c2a34bbabb4934]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:27.331326 2256 tablet_service.cc:1467] Processing CreateTablet for tablet a04dd133580543b186f9346867107b1c (DEFAULT_TABLE table=test-table [id=3b43c41458bc4e8690c2a34bbabb4934]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:27.331632 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04dd133580543b186f9346867107b1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:27.331633 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04dd133580543b186f9346867107b1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:27.335497 2625 tablet_bootstrap.cc:492] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c: Bootstrap starting.
I20260926 09:27:27.335762 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.326398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34015 (local address 127.0.0.1:58684)
0926 09:27:27.326975 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:27.326986 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:27.327041 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:27.327201 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:27.327202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:27.327203 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:27.327236 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:27.327237 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.328095 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.328096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:27.329429 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:27.329432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.330435 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.330437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:27.335508 (+ 5071us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.335515 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 09:27:27.335638 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 09:27:27.335694 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":26,"thread_start_us":67,"threads_started":1}
I20260926 09:27:27.336050 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.326501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58684 (local address 127.0.98.3:34015)
0926 09:27:27.326826 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:27.326830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:27.327055 (+ 225us) server_negotiation.cc:407] Connection header received
0926 09:27:27.327102 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:27.327104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:27.327120 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:27.327203 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:27.327316 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.327997 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.329535 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.330270 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.330343 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.330397 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:27.335903 (+ 5506us) server_negotiation.cc:299] Negotiation successful
0926 09:27:27.335962 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":84,"threads_started":1}
I20260926 09:27:27.336295 2625 tablet_bootstrap.cc:654] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:27.336632 2625 log.cc:824] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:27.336763 2536 tablet_service.cc:1467] Processing CreateTablet for tablet a04dd133580543b186f9346867107b1c (DEFAULT_TABLE table=test-table [id=3b43c41458bc4e8690c2a34bbabb4934]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:27.337004 2626 tablet_bootstrap.cc:492] T a04dd133580543b186f9346867107b1c P 519583d12f9c4e7bac953975b7c7c6ec: Bootstrap starting.
I20260926 09:27:27.337116 2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04dd133580543b186f9346867107b1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:27.337644 2626 tablet_bootstrap.cc:654] T a04dd133580543b186f9346867107b1c P 519583d12f9c4e7bac953975b7c7c6ec: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:27.337973 2626 log.cc:824] T a04dd133580543b186f9346867107b1c P 519583d12f9c4e7bac953975b7c7c6ec: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:27.337997 2625 tablet_bootstrap.cc:492] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c: No bootstrap required, opened a new log
I20260926 09:27:27.338063 2625 ts_tablet_manager.cc:1397] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 09:27:27.338761 2626 tablet_bootstrap.cc:492] T a04dd133580543b186f9346867107b1c P 519583d12f9c4e7bac953975b7c7c6ec: No bootstrap required, opened a new log
I20260926 09:27:27.338845 2626 ts_tablet_manager.cc:1397] T a04dd133580543b186f9346867107b1c P 519583d12f9c4e7bac953975b7c7c6ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:27:27.339932 2625 raft_consensus.cc:348] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.340090 2625 raft_consensus.cc:374] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:27.340132 2625 raft_consensus.cc:729] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712dcc9dcc54da19007ae2c1832554c, State: Initialized, Role: FOLLOWER
I20260926 09:27:27.340263 2625 consensus_queue.cc:260] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.340557 2629 tablet_bootstrap.cc:492] T a04dd133580543b186f9346867107b1c P eb0bc8efbdfd4537a1ae503077a60c4d: Bootstrap starting.
I20260926 09:27:27.340634 2625 ts_tablet_manager.cc:1428] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:27:27.340768 2462 heartbeater.cc:503] Master 127.0.98.62:44869 was elected leader, sending a full tablet report...
I20260926 09:27:27.340955 2626 raft_consensus.cc:348] T a04dd133580543b186f9346867107b1c P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.341135 2626 raft_consensus.cc:374] T a04dd133580543b186f9346867107b1c P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:27.341181 2626 raft_consensus.cc:729] T a04dd133580543b186f9346867107b1c P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519583d12f9c4e7bac953975b7c7c6ec, State: Initialized, Role: FOLLOWER
I20260926 09:27:27.341295 2626 consensus_queue.cc:260] T a04dd133580543b186f9346867107b1c P 519583d12f9c4e7bac953975b7c7c6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.341560 2629 tablet_bootstrap.cc:654] T a04dd133580543b186f9346867107b1c P eb0bc8efbdfd4537a1ae503077a60c4d: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:27.341552 2626 ts_tablet_manager.cc:1428] T a04dd133580543b186f9346867107b1c P 519583d12f9c4e7bac953975b7c7c6ec: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:27:27.341692 2322 heartbeater.cc:503] Master 127.0.98.62:44869 was elected leader, sending a full tablet report...
I20260926 09:27:27.341931 2629 log.cc:824] T a04dd133580543b186f9346867107b1c P eb0bc8efbdfd4537a1ae503077a60c4d: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:27.342585 2629 tablet_bootstrap.cc:492] T a04dd133580543b186f9346867107b1c P eb0bc8efbdfd4537a1ae503077a60c4d: No bootstrap required, opened a new log
I20260926 09:27:27.342638 2629 ts_tablet_manager.cc:1397] T a04dd133580543b186f9346867107b1c P eb0bc8efbdfd4537a1ae503077a60c4d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:27:27.344079 2629 raft_consensus.cc:348] T a04dd133580543b186f9346867107b1c P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.344198 2629 raft_consensus.cc:374] T a04dd133580543b186f9346867107b1c P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:27.344219 2629 raft_consensus.cc:729] T a04dd133580543b186f9346867107b1c P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb0bc8efbdfd4537a1ae503077a60c4d, State: Initialized, Role: FOLLOWER
I20260926 09:27:27.344344 2629 consensus_queue.cc:260] T a04dd133580543b186f9346867107b1c P eb0bc8efbdfd4537a1ae503077a60c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.344597 2629 ts_tablet_manager.cc:1428] T a04dd133580543b186f9346867107b1c P eb0bc8efbdfd4537a1ae503077a60c4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:27:27.344658 2602 heartbeater.cc:503] Master 127.0.98.62:44869 was elected leader, sending a full tablet report...
W20260926 09:27:27.382943 2463 tablet.cc:2369] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:27:27.475721 2323 tablet.cc:2369] T a04dd133580543b186f9346867107b1c P 519583d12f9c4e7bac953975b7c7c6ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:27:27.534022 2603 tablet.cc:2369] T a04dd133580543b186f9346867107b1c P eb0bc8efbdfd4537a1ae503077a60c4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:27:27.638226 2630 raft_consensus.cc:482] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:27:27.638347 2630 raft_consensus.cc:504] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.638653 2630 leader_election.cc:288] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 519583d12f9c4e7bac953975b7c7c6ec (127.0.98.1:34803), eb0bc8efbdfd4537a1ae503077a60c4d (127.0.98.3:34015)
I20260926 09:27:27.641650 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.638970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37369 (local address 127.0.98.1:34803)
0926 09:27:27.639048 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:27.639050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:27.639171 (+ 121us) server_negotiation.cc:407] Connection header received
0926 09:27:27.639236 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:27.639237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:27.639251 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:27.639322 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:27.639493 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.639970 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.640920 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.641275 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.641359 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.641377 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:27.641524 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 09:27:27.641561 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 09:27:27.641660 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.638909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34803 (local address 127.0.98.2:37369)
0926 09:27:27.639059 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:27.639070 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:27.639170 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:27.639335 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:27.639336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:27.639338 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:27.639419 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:27.639421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.640062 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.640063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:27.640818 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:27.640820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.641360 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.641361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:27.641438 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.641442 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:27:27.641518 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 09:27:27.641559 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":101,"threads_started":1}
I20260926 09:27:27.641772 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.639273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51081 (local address 127.0.98.3:34015)
0926 09:27:27.639336 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:27.639337 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:27.639346 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:27:27.639398 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:27.639399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:27.639410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:27.639455 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:27.639622 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.640100 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.641012 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.641371 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.641470 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.641479 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:27.641668 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 09:27:27.641705 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 09:27:27.641786 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.639153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34015 (local address 127.0.98.2:51081)
0926 09:27:27.639286 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:27.639295 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:27.639337 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:27.639471 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:27.639472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:27.639473 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:27.639544 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:27.639546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.640187 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.640188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:27.640921 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:27.640923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.641477 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.641478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:27.641554 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.641558 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:27:27.641660 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 09:27:27.641706 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
I20260926 09:27:27.642063 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04dd133580543b186f9346867107b1c" candidate_uuid: "4712dcc9dcc54da19007ae2c1832554c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" is_pre_election: true
I20260926 09:27:27.642159 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04dd133580543b186f9346867107b1c" candidate_uuid: "4712dcc9dcc54da19007ae2c1832554c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519583d12f9c4e7bac953975b7c7c6ec" is_pre_election: true
I20260926 09:27:27.642233 2556 raft_consensus.cc:2454] T a04dd133580543b186f9346867107b1c P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712dcc9dcc54da19007ae2c1832554c in term 0.
I20260926 09:27:27.642339 2276 raft_consensus.cc:2454] T a04dd133580543b186f9346867107b1c P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712dcc9dcc54da19007ae2c1832554c in term 0.
I20260926 09:27:27.642477 2347 leader_election.cc:302] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712dcc9dcc54da19007ae2c1832554c, eb0bc8efbdfd4537a1ae503077a60c4d; no voters:
I20260926 09:27:27.642613 2630 raft_consensus.cc:2781] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:27:27.642704 2630 raft_consensus.cc:482] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:27:27.642742 2630 raft_consensus.cc:3037] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:27.643453 2630 raft_consensus.cc:504] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.643591 2630 leader_election.cc:288] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c [CANDIDATE]: Term 1 election: Requested vote from peers 519583d12f9c4e7bac953975b7c7c6ec (127.0.98.1:34803), eb0bc8efbdfd4537a1ae503077a60c4d (127.0.98.3:34015)
I20260926 09:27:27.643747 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04dd133580543b186f9346867107b1c" candidate_uuid: "4712dcc9dcc54da19007ae2c1832554c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519583d12f9c4e7bac953975b7c7c6ec"
I20260926 09:27:27.643818 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04dd133580543b186f9346867107b1c" candidate_uuid: "4712dcc9dcc54da19007ae2c1832554c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d"
I20260926 09:27:27.643872 2276 raft_consensus.cc:3037] T a04dd133580543b186f9346867107b1c P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:27.643918 2556 raft_consensus.cc:3037] T a04dd133580543b186f9346867107b1c P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:27.644711 2276 raft_consensus.cc:2454] T a04dd133580543b186f9346867107b1c P 519583d12f9c4e7bac953975b7c7c6ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712dcc9dcc54da19007ae2c1832554c in term 1.
I20260926 09:27:27.644711 2556 raft_consensus.cc:2454] T a04dd133580543b186f9346867107b1c P eb0bc8efbdfd4537a1ae503077a60c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712dcc9dcc54da19007ae2c1832554c in term 1.
I20260926 09:27:27.644914 2347 leader_election.cc:302] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712dcc9dcc54da19007ae2c1832554c, eb0bc8efbdfd4537a1ae503077a60c4d; no voters:
I20260926 09:27:27.644991 2630 raft_consensus.cc:2781] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:27.645103 2630 raft_consensus.cc:686] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c [term 1 LEADER]: Becoming Leader. State: Replica: 4712dcc9dcc54da19007ae2c1832554c, State: Running, Role: LEADER
I20260926 09:27:27.645217 2630 consensus_queue.cc:237] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.646042 2118 catalog_manager.cc:5495] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4712dcc9dcc54da19007ae2c1832554c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4712dcc9dcc54da19007ae2c1832554c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } health_report { overall_health: UNKNOWN } } }
Sep 26 09:27:27 dist-test-slave-dfzm krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:27.708922 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.702275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52384 (local address 127.0.98.62:44869)
0926 09:27:27.702359 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:27.702362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:27.702476 (+ 114us) server_negotiation.cc:407] Connection header received
0926 09:27:27.702558 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:27.702559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:27.702575 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:27.702637 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:27.703108 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.703584 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.704263 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.704442 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.704513 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.706583 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:27.706584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:27.706585 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:27.706591 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:27.707733 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:27.708110 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:27.708111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:27.708111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:27.708176 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:27.708375 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:27.708376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:27.708376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:27.708480 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:27.708552 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:27.708702 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 09:27:27.708787 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 09:27:27.708943 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.702208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44869 (local address 127.0.0.1:52384)
0926 09:27:27.702390 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:27.702411 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:27.702496 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:27.702641 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:27.702641 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:27.702978 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:27.703021 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:27.703023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.703663 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.703664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:27.704181 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:27.704182 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.704521 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.704522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:27.704582 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.705123 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:27.705129 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:27.706494 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:27.707826 (+ 1332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:27.707833 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:27.707836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:27.708022 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:27.708241 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:27.708242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:27.708243 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:27.708299 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:27.708544 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:27.708551 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:27.708595 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 09:27:27.708681 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 09:27:27.708851 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":77,"thread_start_us":97,"threads_started":1}
I20260926 09:27:27.710992 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52384:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 09:27:27.711216 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:27:27.712772 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 20427dd08d8543458ed7d80d447de740 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5dd2f50621c5428ca426b1d7869c9206]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:27:27.712786 2536 tablet_service.cc:1467] Processing CreateTablet for tablet 20427dd08d8543458ed7d80d447de740 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5dd2f50621c5428ca426b1d7869c9206]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:27:27.712826 2396 tablet_service.cc:1467] Processing CreateTablet for tablet 20427dd08d8543458ed7d80d447de740 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5dd2f50621c5428ca426b1d7869c9206]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:27:27.712961 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20427dd08d8543458ed7d80d447de740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:27.712961 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20427dd08d8543458ed7d80d447de740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:27.712970 2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20427dd08d8543458ed7d80d447de740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:27.714445 2626 tablet_bootstrap.cc:492] T 20427dd08d8543458ed7d80d447de740 P 519583d12f9c4e7bac953975b7c7c6ec: Bootstrap starting.
I20260926 09:27:27.714458 2629 tablet_bootstrap.cc:492] T 20427dd08d8543458ed7d80d447de740 P eb0bc8efbdfd4537a1ae503077a60c4d: Bootstrap starting.
I20260926 09:27:27.714812 2626 tablet_bootstrap.cc:654] T 20427dd08d8543458ed7d80d447de740 P 519583d12f9c4e7bac953975b7c7c6ec: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:27.714985 2629 tablet_bootstrap.cc:654] T 20427dd08d8543458ed7d80d447de740 P eb0bc8efbdfd4537a1ae503077a60c4d: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:27.715359 2625 tablet_bootstrap.cc:492] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c: Bootstrap starting.
I20260926 09:27:27.715518 2626 tablet_bootstrap.cc:492] T 20427dd08d8543458ed7d80d447de740 P 519583d12f9c4e7bac953975b7c7c6ec: No bootstrap required, opened a new log
I20260926 09:27:27.715586 2626 ts_tablet_manager.cc:1397] T 20427dd08d8543458ed7d80d447de740 P 519583d12f9c4e7bac953975b7c7c6ec: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:27.715699 2625 tablet_bootstrap.cc:654] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:27.715764 2626 raft_consensus.cc:348] T 20427dd08d8543458ed7d80d447de740 P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.715842 2626 raft_consensus.cc:374] T 20427dd08d8543458ed7d80d447de740 P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:27.715868 2626 raft_consensus.cc:729] T 20427dd08d8543458ed7d80d447de740 P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519583d12f9c4e7bac953975b7c7c6ec, State: Initialized, Role: FOLLOWER
I20260926 09:27:27.715930 2626 consensus_queue.cc:260] T 20427dd08d8543458ed7d80d447de740 P 519583d12f9c4e7bac953975b7c7c6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.716035 2626 ts_tablet_manager.cc:1428] T 20427dd08d8543458ed7d80d447de740 P 519583d12f9c4e7bac953975b7c7c6ec: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:27.716177 2631 tablet_replica.cc:439] T 20427dd08d8543458ed7d80d447de740 P 519583d12f9c4e7bac953975b7c7c6ec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } } }
I20260926 09:27:27.716280 2631 tablet_replica.cc:442] T 20427dd08d8543458ed7d80d447de740 P 519583d12f9c4e7bac953975b7c7c6ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:27:27.716344 2629 tablet_bootstrap.cc:492] T 20427dd08d8543458ed7d80d447de740 P eb0bc8efbdfd4537a1ae503077a60c4d: No bootstrap required, opened a new log
I20260926 09:27:27.716377 2629 ts_tablet_manager.cc:1397] T 20427dd08d8543458ed7d80d447de740 P eb0bc8efbdfd4537a1ae503077a60c4d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:27:27.716481 2629 raft_consensus.cc:348] T 20427dd08d8543458ed7d80d447de740 P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.716566 2629 raft_consensus.cc:374] T 20427dd08d8543458ed7d80d447de740 P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:27.716588 2629 raft_consensus.cc:729] T 20427dd08d8543458ed7d80d447de740 P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb0bc8efbdfd4537a1ae503077a60c4d, State: Initialized, Role: FOLLOWER
I20260926 09:27:27.716640 2629 consensus_queue.cc:260] T 20427dd08d8543458ed7d80d447de740 P eb0bc8efbdfd4537a1ae503077a60c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.716749 2629 ts_tablet_manager.cc:1428] T 20427dd08d8543458ed7d80d447de740 P eb0bc8efbdfd4537a1ae503077a60c4d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:27.716991 2625 tablet_bootstrap.cc:492] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c: No bootstrap required, opened a new log
I20260926 09:27:27.717041 2625 ts_tablet_manager.cc:1397] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:27:27.717015 2633 tablet_replica.cc:439] T 20427dd08d8543458ed7d80d447de740 P eb0bc8efbdfd4537a1ae503077a60c4d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } } }
I20260926 09:27:27.717095 2633 tablet_replica.cc:442] T 20427dd08d8543458ed7d80d447de740 P eb0bc8efbdfd4537a1ae503077a60c4d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:27:27.717195 2625 raft_consensus.cc:348] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.717281 2625 raft_consensus.cc:374] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:27.717314 2625 raft_consensus.cc:729] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712dcc9dcc54da19007ae2c1832554c, State: Initialized, Role: FOLLOWER
I20260926 09:27:27.717365 2625 consensus_queue.cc:260] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.717453 2625 ts_tablet_manager.cc:1428] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:27.717502 2630 tablet_replica.cc:439] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } } }
I20260926 09:27:27.717615 2630 tablet_replica.cc:442] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:27:27.767619 2630 raft_consensus.cc:482] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:27:27.767711 2630 raft_consensus.cc:504] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.767880 2630 leader_election.cc:288] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 519583d12f9c4e7bac953975b7c7c6ec (127.0.98.1:34803), eb0bc8efbdfd4537a1ae503077a60c4d (127.0.98.3:34015)
I20260926 09:27:27.768121 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20427dd08d8543458ed7d80d447de740" candidate_uuid: "4712dcc9dcc54da19007ae2c1832554c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519583d12f9c4e7bac953975b7c7c6ec" is_pre_election: true
I20260926 09:27:27.768205 2276 raft_consensus.cc:2454] T 20427dd08d8543458ed7d80d447de740 P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712dcc9dcc54da19007ae2c1832554c in term 0.
I20260926 09:27:27.768123 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20427dd08d8543458ed7d80d447de740" candidate_uuid: "4712dcc9dcc54da19007ae2c1832554c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" is_pre_election: true
I20260926 09:27:27.768246 2556 raft_consensus.cc:2454] T 20427dd08d8543458ed7d80d447de740 P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712dcc9dcc54da19007ae2c1832554c in term 0.
I20260926 09:27:27.768364 2347 leader_election.cc:302] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712dcc9dcc54da19007ae2c1832554c, 519583d12f9c4e7bac953975b7c7c6ec; no voters:
I20260926 09:27:27.768471 2630 raft_consensus.cc:2781] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:27:27.768509 2630 raft_consensus.cc:482] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:27:27.768539 2630 raft_consensus.cc:3037] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:27.769157 2630 raft_consensus.cc:504] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.769280 2630 leader_election.cc:288] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c [CANDIDATE]: Term 1 election: Requested vote from peers 519583d12f9c4e7bac953975b7c7c6ec (127.0.98.1:34803), eb0bc8efbdfd4537a1ae503077a60c4d (127.0.98.3:34015)
I20260926 09:27:27.769466 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20427dd08d8543458ed7d80d447de740" candidate_uuid: "4712dcc9dcc54da19007ae2c1832554c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519583d12f9c4e7bac953975b7c7c6ec"
I20260926 09:27:27.769553 2276 raft_consensus.cc:3037] T 20427dd08d8543458ed7d80d447de740 P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:27.769526 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20427dd08d8543458ed7d80d447de740" candidate_uuid: "4712dcc9dcc54da19007ae2c1832554c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d"
I20260926 09:27:27.769649 2556 raft_consensus.cc:3037] T 20427dd08d8543458ed7d80d447de740 P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:27.770241 2276 raft_consensus.cc:2454] T 20427dd08d8543458ed7d80d447de740 P 519583d12f9c4e7bac953975b7c7c6ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712dcc9dcc54da19007ae2c1832554c in term 1.
I20260926 09:27:27.770323 2556 raft_consensus.cc:2454] T 20427dd08d8543458ed7d80d447de740 P eb0bc8efbdfd4537a1ae503077a60c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712dcc9dcc54da19007ae2c1832554c in term 1.
I20260926 09:27:27.770404 2347 leader_election.cc:302] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712dcc9dcc54da19007ae2c1832554c, 519583d12f9c4e7bac953975b7c7c6ec; no voters:
I20260926 09:27:27.770515 2630 raft_consensus.cc:2781] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:27.770565 2630 raft_consensus.cc:686] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c [term 1 LEADER]: Becoming Leader. State: Replica: 4712dcc9dcc54da19007ae2c1832554c, State: Running, Role: LEADER
I20260926 09:27:27.770622 2630 consensus_queue.cc:237] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.770936 2636 tablet_replica.cc:439] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c: TxnStatusTablet state changed. Reason: New leader 4712dcc9dcc54da19007ae2c1832554c. Latest consensus state: current_term: 1 leader_uuid: "4712dcc9dcc54da19007ae2c1832554c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } } }
I20260926 09:27:27.771034 2636 tablet_replica.cc:442] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:27:27.771245 2648 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 09:27:27.771248 2119 catalog_manager.cc:5495] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4712dcc9dcc54da19007ae2c1832554c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4712dcc9dcc54da19007ae2c1832554c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } health_report { overall_health: UNKNOWN } } }
I20260926 09:27:27.771395 2648 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 09:27:27.778951 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.776270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50138 (local address 127.0.98.2:36885)
0926 09:27:27.776326 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:27.776328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:27.776337 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:27:27.776420 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:27.776421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:27.776435 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:27.776492 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:27:27.776611 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.777072 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.778182 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.778364 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.778441 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.778620 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:27:27.778705 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:27:27.778761 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:27.778835 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 09:27:27.778873 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 09:27:27.778972 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.776205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36885 (local address 127.0.0.1:50138)
0926 09:27:27.776289 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:27.776301 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:27.776343 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:27.776497 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:27.776498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:27.776499 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:27:27.776550 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:27.776551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.777151 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.777152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:27.778102 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:27.778105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.778444 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.778446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:27.778515 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.778532 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:27:27.778767 (+ 235us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:27:27.778775 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 09:27:27.778832 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 09:27:27.778892 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 09:27:27.780977 2556 raft_consensus.cc:1261] T 20427dd08d8543458ed7d80d447de740 P eb0bc8efbdfd4537a1ae503077a60c4d [term 1 FOLLOWER]: Refusing update from remote peer 4712dcc9dcc54da19007ae2c1832554c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:27.781401 2276 raft_consensus.cc:1261] T 20427dd08d8543458ed7d80d447de740 P 519583d12f9c4e7bac953975b7c7c6ec [term 1 FOLLOWER]: Refusing update from remote peer 4712dcc9dcc54da19007ae2c1832554c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:27.781399 2630 consensus_queue.cc:1035] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:27.781735 2630 consensus_queue.cc:1035] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c [LEADER]: Connected to new peer: Peer: permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:27.782723 2633 tablet_replica.cc:439] T 20427dd08d8543458ed7d80d447de740 P eb0bc8efbdfd4537a1ae503077a60c4d: TxnStatusTablet state changed. Reason: New leader 4712dcc9dcc54da19007ae2c1832554c. Latest consensus state: current_term: 1 leader_uuid: "4712dcc9dcc54da19007ae2c1832554c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } } }
I20260926 09:27:27.782855 2633 tablet_replica.cc:442] T 20427dd08d8543458ed7d80d447de740 P eb0bc8efbdfd4537a1ae503077a60c4d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:27:27.783883 2631 tablet_replica.cc:439] T 20427dd08d8543458ed7d80d447de740 P 519583d12f9c4e7bac953975b7c7c6ec: TxnStatusTablet state changed. Reason: New leader 4712dcc9dcc54da19007ae2c1832554c. Latest consensus state: current_term: 1 leader_uuid: "4712dcc9dcc54da19007ae2c1832554c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } } }
I20260926 09:27:27.783975 2631 tablet_replica.cc:442] T 20427dd08d8543458ed7d80d447de740 P 519583d12f9c4e7bac953975b7c7c6ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:27:27.784574 2630 tablet_replica.cc:439] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4712dcc9dcc54da19007ae2c1832554c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } } }
I20260926 09:27:27.784668 2630 tablet_replica.cc:442] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:27:27.784741 2651 mvcc.cc:205] Tried to move back new op lower bound from 7333539216506986496 to 7333539216468774912. Current Snapshot: MvccSnapshot[applied={T|T < 7333539216506986496}]
I20260926 09:27:27.785868 2636 tablet_replica.cc:439] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4712dcc9dcc54da19007ae2c1832554c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } } }
I20260926 09:27:27.785962 2636 tablet_replica.cc:442] T 20427dd08d8543458ed7d80d447de740 P 4712dcc9dcc54da19007ae2c1832554c: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:27:27.786136 2633 tablet_replica.cc:439] T 20427dd08d8543458ed7d80d447de740 P eb0bc8efbdfd4537a1ae503077a60c4d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4712dcc9dcc54da19007ae2c1832554c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } } }
I20260926 09:27:27.786237 2633 tablet_replica.cc:442] T 20427dd08d8543458ed7d80d447de740 P eb0bc8efbdfd4537a1ae503077a60c4d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:27:27.786557 2631 tablet_replica.cc:439] T 20427dd08d8543458ed7d80d447de740 P 519583d12f9c4e7bac953975b7c7c6ec: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4712dcc9dcc54da19007ae2c1832554c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } } }
I20260926 09:27:27.786648 2631 tablet_replica.cc:442] T 20427dd08d8543458ed7d80d447de740 P 519583d12f9c4e7bac953975b7c7c6ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:27:27.787084 2654 mvcc.cc:205] Tried to move back new op lower bound from 7333539216506986496 to 7333539216468774912. Current Snapshot: MvccSnapshot[applied={T|T < 7333539216506986496}]
I20260926 09:27:27.790365 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.787631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50154 (local address 127.0.98.2:36885)
0926 09:27:27.787700 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:27.787702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:27.787713 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:27:27.787733 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:27.787734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:27.787749 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:27.787809 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:27:27.787974 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.788443 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.789687 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.789957 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.790040 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.790191 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:27:27.790217 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:27:27.790249 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:27.790301 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 09:27:27.790319 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 09:27:27.795436 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.792762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34667 (local address 127.0.98.2:36885)
0926 09:27:27.792802 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:27.792804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:27.792939 (+ 135us) server_negotiation.cc:407] Connection header received
0926 09:27:27.792963 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:27.792964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:27.792973 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:27.793036 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:27:27.793219 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.793952 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.794672 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.794971 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.795042 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.795210 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:27:27.795240 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:27:27.795292 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:27.795337 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 09:27:27.795353 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:27:27.795513 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:27.792708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36885 (local address 127.0.98.2:34667)
0926 09:27:27.792858 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:27.792868 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:27.792930 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:27.793051 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:27.793052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:27.793067 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:27:27.793143 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:27.793144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.794022 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.794022 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:27.794599 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:27.794600 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:27.795064 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:27.795065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:27.795112 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:27.795123 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:27:27.795298 (+ 175us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:27:27.795300 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:27:27.795334 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 09:27:27.795365 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":95,"threads_started":1}
I20260926 09:27:27.797709 2276 raft_consensus.cc:1261] T a04dd133580543b186f9346867107b1c P 519583d12f9c4e7bac953975b7c7c6ec [term 1 FOLLOWER]: Refusing update from remote peer 4712dcc9dcc54da19007ae2c1832554c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:27.797715 2556 raft_consensus.cc:1261] T a04dd133580543b186f9346867107b1c P eb0bc8efbdfd4537a1ae503077a60c4d [term 1 FOLLOWER]: Refusing update from remote peer 4712dcc9dcc54da19007ae2c1832554c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:27.797897 2658 consensus_queue.cc:1035] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c [LEADER]: Connected to new peer: Peer: permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:27.798194 2664 consensus_queue.cc:1035] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:27.799913 2652 mvcc.cc:205] Tried to move back new op lower bound from 7333539216577654784 to 7333539215955410944. Current Snapshot: MvccSnapshot[applied={T|T < 7333539216577654784}]
I20260926 09:27:27.812033 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:27.812136 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:27.812840 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f: Sending DeleteTablet for 3 replicas of tablet a04dd133580543b186f9346867107b1c
I20260926 09:27:27.813189 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet a04dd133580543b186f9346867107b1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53432
I20260926 09:27:27.813194 2536 tablet_service.cc:1514] Processing DeleteTablet for tablet a04dd133580543b186f9346867107b1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58684
I20260926 09:27:27.813402 2396 tablet_service.cc:1514] Processing DeleteTablet for tablet a04dd133580543b186f9346867107b1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50134
I20260926 09:27:27.813650 2670 tablet_replica.cc:330] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c: stopping tablet replica
I20260926 09:27:27.813755 2670 raft_consensus.cc:2229] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:27:27.813865 2670 raft_consensus.cc:2258] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:27.813922 2669 tablet_replica.cc:330] T a04dd133580543b186f9346867107b1c P 519583d12f9c4e7bac953975b7c7c6ec: stopping tablet replica
I20260926 09:27:27.813966 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 09:27:27.814103 2669 raft_consensus.cc:2229] T a04dd133580543b186f9346867107b1c P 519583d12f9c4e7bac953975b7c7c6ec [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 09:27:27.814105 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:27:27.814185 2669 raft_consensus.cc:2258] T a04dd133580543b186f9346867107b1c P 519583d12f9c4e7bac953975b7c7c6ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:27.814388 2668 tablet_replica.cc:330] T a04dd133580543b186f9346867107b1c P eb0bc8efbdfd4537a1ae503077a60c4d: stopping tablet replica
I20260926 09:27:27.814482 2668 raft_consensus.cc:2229] T a04dd133580543b186f9346867107b1c P eb0bc8efbdfd4537a1ae503077a60c4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:27.814558 2668 raft_consensus.cc:2258] T a04dd133580543b186f9346867107b1c P eb0bc8efbdfd4537a1ae503077a60c4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:27.814985 2668 ts_tablet_manager.cc:1905] T a04dd133580543b186f9346867107b1c P eb0bc8efbdfd4537a1ae503077a60c4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:27.815269 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 69ccaca5fe2e4cf3995a1691f8653b22 (DEFAULT_TABLE table=test-table [id=ac172d174d6c482e891b1204013ac0ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:27.815318 2396 tablet_service.cc:1467] Processing CreateTablet for tablet 69ccaca5fe2e4cf3995a1691f8653b22 (DEFAULT_TABLE table=test-table [id=ac172d174d6c482e891b1204013ac0ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:27.815387 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69ccaca5fe2e4cf3995a1691f8653b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:27.815416 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69ccaca5fe2e4cf3995a1691f8653b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:27.816732 2625 tablet_bootstrap.cc:492] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c: Bootstrap starting.
I20260926 09:27:27.816885 2668 ts_tablet_manager.cc:1918] T a04dd133580543b186f9346867107b1c P eb0bc8efbdfd4537a1ae503077a60c4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:27:27.816951 2668 log.cc:1196] T a04dd133580543b186f9346867107b1c P eb0bc8efbdfd4537a1ae503077a60c4d: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/a04dd133580543b186f9346867107b1c
I20260926 09:27:27.817176 2625 tablet_bootstrap.cc:654] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:27.817193 2668 ts_tablet_manager.cc:1939] T a04dd133580543b186f9346867107b1c P eb0bc8efbdfd4537a1ae503077a60c4d: Deleting consensus metadata
I20260926 09:27:27.817524 2626 tablet_bootstrap.cc:492] T 69ccaca5fe2e4cf3995a1691f8653b22 P 519583d12f9c4e7bac953975b7c7c6ec: Bootstrap starting.
I20260926 09:27:27.817574 2103 catalog_manager.cc:4841] TS eb0bc8efbdfd4537a1ae503077a60c4d (127.0.98.3:34015): tablet a04dd133580543b186f9346867107b1c (table test-table [id=3b43c41458bc4e8690c2a34bbabb4934]) successfully deleted
I20260926 09:27:27.817656 2536 tablet_service.cc:1467] Processing CreateTablet for tablet 69ccaca5fe2e4cf3995a1691f8653b22 (DEFAULT_TABLE table=test-table [id=ac172d174d6c482e891b1204013ac0ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:27.817761 2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69ccaca5fe2e4cf3995a1691f8653b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:27.818017 2625 tablet_bootstrap.cc:492] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c: No bootstrap required, opened a new log
I20260926 09:27:27.818068 2625 ts_tablet_manager.cc:1397] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:27.818198 2625 raft_consensus.cc:348] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.818269 2625 raft_consensus.cc:374] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:27.818289 2625 raft_consensus.cc:729] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712dcc9dcc54da19007ae2c1832554c, State: Initialized, Role: FOLLOWER
I20260926 09:27:27.818328 2625 consensus_queue.cc:260] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.818413 2625 ts_tablet_manager.cc:1428] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:27.819173 2629 tablet_bootstrap.cc:492] T 69ccaca5fe2e4cf3995a1691f8653b22 P eb0bc8efbdfd4537a1ae503077a60c4d: Bootstrap starting.
I20260926 09:27:27.819502 2626 tablet_bootstrap.cc:654] T 69ccaca5fe2e4cf3995a1691f8653b22 P 519583d12f9c4e7bac953975b7c7c6ec: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:27.819545 2629 tablet_bootstrap.cc:654] T 69ccaca5fe2e4cf3995a1691f8653b22 P eb0bc8efbdfd4537a1ae503077a60c4d: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:27.820108 2626 tablet_bootstrap.cc:492] T 69ccaca5fe2e4cf3995a1691f8653b22 P 519583d12f9c4e7bac953975b7c7c6ec: No bootstrap required, opened a new log
I20260926 09:27:27.820163 2626 ts_tablet_manager.cc:1397] T 69ccaca5fe2e4cf3995a1691f8653b22 P 519583d12f9c4e7bac953975b7c7c6ec: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 09:27:27.820318 2626 raft_consensus.cc:348] T 69ccaca5fe2e4cf3995a1691f8653b22 P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.820392 2626 raft_consensus.cc:374] T 69ccaca5fe2e4cf3995a1691f8653b22 P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:27.820421 2626 raft_consensus.cc:729] T 69ccaca5fe2e4cf3995a1691f8653b22 P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519583d12f9c4e7bac953975b7c7c6ec, State: Initialized, Role: FOLLOWER
I20260926 09:27:27.820467 2626 consensus_queue.cc:260] T 69ccaca5fe2e4cf3995a1691f8653b22 P 519583d12f9c4e7bac953975b7c7c6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.820554 2626 ts_tablet_manager.cc:1428] T 69ccaca5fe2e4cf3995a1691f8653b22 P 519583d12f9c4e7bac953975b7c7c6ec: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:27.820744 2669 ts_tablet_manager.cc:1905] T a04dd133580543b186f9346867107b1c P 519583d12f9c4e7bac953975b7c7c6ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:27.821725 2670 ts_tablet_manager.cc:1905] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:27.822374 2669 ts_tablet_manager.cc:1918] T a04dd133580543b186f9346867107b1c P 519583d12f9c4e7bac953975b7c7c6ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:27:27.822460 2669 log.cc:1196] T a04dd133580543b186f9346867107b1c P 519583d12f9c4e7bac953975b7c7c6ec: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-0/wal/wals/a04dd133580543b186f9346867107b1c
I20260926 09:27:27.822695 2669 ts_tablet_manager.cc:1939] T a04dd133580543b186f9346867107b1c P 519583d12f9c4e7bac953975b7c7c6ec: Deleting consensus metadata
I20260926 09:27:27.823103 2103 catalog_manager.cc:4841] TS 519583d12f9c4e7bac953975b7c7c6ec (127.0.98.1:34803): tablet a04dd133580543b186f9346867107b1c (table test-table [id=3b43c41458bc4e8690c2a34bbabb4934]) successfully deleted
I20260926 09:27:27.823218 2670 ts_tablet_manager.cc:1918] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:27:27.823288 2670 log.cc:1196] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-1/wal/wals/a04dd133580543b186f9346867107b1c
I20260926 09:27:27.823545 2670 ts_tablet_manager.cc:1939] T a04dd133580543b186f9346867107b1c P 4712dcc9dcc54da19007ae2c1832554c: Deleting consensus metadata
I20260926 09:27:27.823900 2106 catalog_manager.cc:4841] TS 4712dcc9dcc54da19007ae2c1832554c (127.0.98.2:36885): tablet a04dd133580543b186f9346867107b1c (table test-table [id=3b43c41458bc4e8690c2a34bbabb4934]) successfully deleted
I20260926 09:27:27.824180 2629 tablet_bootstrap.cc:492] T 69ccaca5fe2e4cf3995a1691f8653b22 P eb0bc8efbdfd4537a1ae503077a60c4d: No bootstrap required, opened a new log
I20260926 09:27:27.824230 2629 ts_tablet_manager.cc:1397] T 69ccaca5fe2e4cf3995a1691f8653b22 P eb0bc8efbdfd4537a1ae503077a60c4d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 09:27:27.824373 2629 raft_consensus.cc:348] T 69ccaca5fe2e4cf3995a1691f8653b22 P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.824481 2629 raft_consensus.cc:374] T 69ccaca5fe2e4cf3995a1691f8653b22 P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:27.824513 2629 raft_consensus.cc:729] T 69ccaca5fe2e4cf3995a1691f8653b22 P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb0bc8efbdfd4537a1ae503077a60c4d, State: Initialized, Role: FOLLOWER
I20260926 09:27:27.824561 2629 consensus_queue.cc:260] T 69ccaca5fe2e4cf3995a1691f8653b22 P eb0bc8efbdfd4537a1ae503077a60c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.824651 2629 ts_tablet_manager.cc:1428] T 69ccaca5fe2e4cf3995a1691f8653b22 P eb0bc8efbdfd4537a1ae503077a60c4d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:27.828176 2658 raft_consensus.cc:482] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:27:27.828240 2658 raft_consensus.cc:504] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.828358 2658 leader_election.cc:288] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 519583d12f9c4e7bac953975b7c7c6ec (127.0.98.1:34803), eb0bc8efbdfd4537a1ae503077a60c4d (127.0.98.3:34015)
I20260926 09:27:27.828538 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69ccaca5fe2e4cf3995a1691f8653b22" candidate_uuid: "4712dcc9dcc54da19007ae2c1832554c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" is_pre_election: true
I20260926 09:27:27.828534 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69ccaca5fe2e4cf3995a1691f8653b22" candidate_uuid: "4712dcc9dcc54da19007ae2c1832554c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519583d12f9c4e7bac953975b7c7c6ec" is_pre_election: true
I20260926 09:27:27.828614 2556 raft_consensus.cc:2454] T 69ccaca5fe2e4cf3995a1691f8653b22 P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712dcc9dcc54da19007ae2c1832554c in term 0.
I20260926 09:27:27.828614 2276 raft_consensus.cc:2454] T 69ccaca5fe2e4cf3995a1691f8653b22 P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712dcc9dcc54da19007ae2c1832554c in term 0.
I20260926 09:27:27.828737 2347 leader_election.cc:302] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712dcc9dcc54da19007ae2c1832554c, 519583d12f9c4e7bac953975b7c7c6ec; no voters:
I20260926 09:27:27.828835 2658 raft_consensus.cc:2781] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:27:27.828886 2658 raft_consensus.cc:482] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:27:27.828907 2658 raft_consensus.cc:3037] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:27.829442 2658 raft_consensus.cc:504] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.829560 2658 leader_election.cc:288] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c [CANDIDATE]: Term 1 election: Requested vote from peers 519583d12f9c4e7bac953975b7c7c6ec (127.0.98.1:34803), eb0bc8efbdfd4537a1ae503077a60c4d (127.0.98.3:34015)
I20260926 09:27:27.829756 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69ccaca5fe2e4cf3995a1691f8653b22" candidate_uuid: "4712dcc9dcc54da19007ae2c1832554c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d"
I20260926 09:27:27.829810 2556 raft_consensus.cc:3037] T 69ccaca5fe2e4cf3995a1691f8653b22 P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:27.830322 2556 raft_consensus.cc:2454] T 69ccaca5fe2e4cf3995a1691f8653b22 P eb0bc8efbdfd4537a1ae503077a60c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712dcc9dcc54da19007ae2c1832554c in term 1.
I20260926 09:27:27.830442 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69ccaca5fe2e4cf3995a1691f8653b22" candidate_uuid: "4712dcc9dcc54da19007ae2c1832554c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519583d12f9c4e7bac953975b7c7c6ec"
I20260926 09:27:27.830502 2347 leader_election.cc:302] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712dcc9dcc54da19007ae2c1832554c, eb0bc8efbdfd4537a1ae503077a60c4d; no voters:
I20260926 09:27:27.830516 2276 raft_consensus.cc:3037] T 69ccaca5fe2e4cf3995a1691f8653b22 P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:27.830577 2658 raft_consensus.cc:2781] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:27.830612 2658 raft_consensus.cc:686] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c [term 1 LEADER]: Becoming Leader. State: Replica: 4712dcc9dcc54da19007ae2c1832554c, State: Running, Role: LEADER
I20260926 09:27:27.830652 2658 consensus_queue.cc:237] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:27.831125 2118 catalog_manager.cc:5495] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4712dcc9dcc54da19007ae2c1832554c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4712dcc9dcc54da19007ae2c1832554c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } health_report { overall_health: UNKNOWN } } }
I20260926 09:27:27.831279 2276 raft_consensus.cc:2454] T 69ccaca5fe2e4cf3995a1691f8653b22 P 519583d12f9c4e7bac953975b7c7c6ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712dcc9dcc54da19007ae2c1832554c in term 1.
I20260926 09:27:27.839692 2556 raft_consensus.cc:1261] T 69ccaca5fe2e4cf3995a1691f8653b22 P eb0bc8efbdfd4537a1ae503077a60c4d [term 1 FOLLOWER]: Refusing update from remote peer 4712dcc9dcc54da19007ae2c1832554c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:27.839840 2276 raft_consensus.cc:1261] T 69ccaca5fe2e4cf3995a1691f8653b22 P 519583d12f9c4e7bac953975b7c7c6ec [term 1 FOLLOWER]: Refusing update from remote peer 4712dcc9dcc54da19007ae2c1832554c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:27.839906 2630 consensus_queue.cc:1035] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:27.839990 2664 consensus_queue.cc:1035] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c [LEADER]: Connected to new peer: Peer: permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 09:27:27.844127 2651 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 09:27:29.361663 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:29.361815 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:29.363065 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f: Sending DeleteTablet for 3 replicas of tablet 69ccaca5fe2e4cf3995a1691f8653b22
I20260926 09:27:29.363330 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet 69ccaca5fe2e4cf3995a1691f8653b22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53432
I20260926 09:27:29.363363 2396 tablet_service.cc:1514] Processing DeleteTablet for tablet 69ccaca5fe2e4cf3995a1691f8653b22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50134
I20260926 09:27:29.363366 2536 tablet_service.cc:1514] Processing DeleteTablet for tablet 69ccaca5fe2e4cf3995a1691f8653b22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58684
I20260926 09:27:29.363588 2698 tablet_replica.cc:330] T 69ccaca5fe2e4cf3995a1691f8653b22 P 519583d12f9c4e7bac953975b7c7c6ec: stopping tablet replica
I20260926 09:27:29.363593 2697 tablet_replica.cc:330] T 69ccaca5fe2e4cf3995a1691f8653b22 P eb0bc8efbdfd4537a1ae503077a60c4d: stopping tablet replica
I20260926 09:27:29.363634 2698 raft_consensus.cc:2229] T 69ccaca5fe2e4cf3995a1691f8653b22 P 519583d12f9c4e7bac953975b7c7c6ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:29.363663 2698 raft_consensus.cc:2258] T 69ccaca5fe2e4cf3995a1691f8653b22 P 519583d12f9c4e7bac953975b7c7c6ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:29.363654 2697 raft_consensus.cc:2229] T 69ccaca5fe2e4cf3995a1691f8653b22 P eb0bc8efbdfd4537a1ae503077a60c4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:29.363857 2698 ts_tablet_manager.cc:1905] T 69ccaca5fe2e4cf3995a1691f8653b22 P 519583d12f9c4e7bac953975b7c7c6ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:29.363853 2696 tablet_replica.cc:330] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c: stopping tablet replica
I20260926 09:27:29.363907 2696 raft_consensus.cc:2229] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:27:29.363963 2696 raft_consensus.cc:2258] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:29.363694 2697 raft_consensus.cc:2258] T 69ccaca5fe2e4cf3995a1691f8653b22 P eb0bc8efbdfd4537a1ae503077a60c4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:29.364228 2696 ts_tablet_manager.cc:1905] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:29.365330 2697 ts_tablet_manager.cc:1905] T 69ccaca5fe2e4cf3995a1691f8653b22 P eb0bc8efbdfd4537a1ae503077a60c4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:29.367424 2697 ts_tablet_manager.cc:1918] T 69ccaca5fe2e4cf3995a1691f8653b22 P eb0bc8efbdfd4537a1ae503077a60c4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 09:27:29.367491 2697 log.cc:1196] T 69ccaca5fe2e4cf3995a1691f8653b22 P eb0bc8efbdfd4537a1ae503077a60c4d: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/69ccaca5fe2e4cf3995a1691f8653b22
I20260926 09:27:29.367729 2697 ts_tablet_manager.cc:1939] T 69ccaca5fe2e4cf3995a1691f8653b22 P eb0bc8efbdfd4537a1ae503077a60c4d: Deleting consensus metadata
I20260926 09:27:29.370064 2103 catalog_manager.cc:4841] TS eb0bc8efbdfd4537a1ae503077a60c4d (127.0.98.3:34015): tablet 69ccaca5fe2e4cf3995a1691f8653b22 (table test-table [id=ac172d174d6c482e891b1204013ac0ab]) successfully deleted
Sep 26 09:27:29 dist-test-slave-dfzm krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414849, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260926 09:27:29.382839 2698 ts_tablet_manager.cc:1918] T 69ccaca5fe2e4cf3995a1691f8653b22 P 519583d12f9c4e7bac953975b7c7c6ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 09:27:29.382839 2696 ts_tablet_manager.cc:1918] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 09:27:29.382929 2698 log.cc:1196] T 69ccaca5fe2e4cf3995a1691f8653b22 P 519583d12f9c4e7bac953975b7c7c6ec: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-0/wal/wals/69ccaca5fe2e4cf3995a1691f8653b22
I20260926 09:27:29.382932 2696 log.cc:1196] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-1/wal/wals/69ccaca5fe2e4cf3995a1691f8653b22
I20260926 09:27:29.383168 2698 ts_tablet_manager.cc:1939] T 69ccaca5fe2e4cf3995a1691f8653b22 P 519583d12f9c4e7bac953975b7c7c6ec: Deleting consensus metadata
I20260926 09:27:29.383234 2696 ts_tablet_manager.cc:1939] T 69ccaca5fe2e4cf3995a1691f8653b22 P 4712dcc9dcc54da19007ae2c1832554c: Deleting consensus metadata
I20260926 09:27:29.384788 2103 catalog_manager.cc:4841] TS 519583d12f9c4e7bac953975b7c7c6ec (127.0.98.1:34803): tablet 69ccaca5fe2e4cf3995a1691f8653b22 (table test-table [id=ac172d174d6c482e891b1204013ac0ab]) successfully deleted
I20260926 09:27:29.384872 2106 catalog_manager.cc:4841] TS 4712dcc9dcc54da19007ae2c1832554c (127.0.98.2:36885): tablet 69ccaca5fe2e4cf3995a1691f8653b22 (table test-table [id=ac172d174d6c482e891b1204013ac0ab]) successfully deleted
Sep 26 09:27:29 dist-test-slave-dfzm krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414849, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:29.391837 2706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:29.383982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52392 (local address 127.0.98.62:44869)
0926 09:27:29.384136 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:29.384140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:29.384205 (+ 65us) server_negotiation.cc:407] Connection header received
0926 09:27:29.384277 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:29.384278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:29.384288 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:29.384354 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:29.385085 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:29.385631 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:29.386544 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:29.386740 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:29.386796 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:29.389212 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:29.389214 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:29.389214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:29.389221 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:29.390667 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:29.390978 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:29.390979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:29.390979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:29.391032 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:29.391331 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:29.391333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:29.391333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:29.391454 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:29.391556 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:29.391659 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 09:27:29.391728 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260926 09:27:29.393447 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:27:29.393643 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:27:29.395056 2536 tablet_service.cc:1467] Processing CreateTablet for tablet b26c50a93eb842b183f5753c69b6795c (DEFAULT_TABLE table=test-table [id=fab73127c4474504af960a7775dc5640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:29.395058 2256 tablet_service.cc:1467] Processing CreateTablet for tablet b26c50a93eb842b183f5753c69b6795c (DEFAULT_TABLE table=test-table [id=fab73127c4474504af960a7775dc5640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:29.395042 2396 tablet_service.cc:1467] Processing CreateTablet for tablet b26c50a93eb842b183f5753c69b6795c (DEFAULT_TABLE table=test-table [id=fab73127c4474504af960a7775dc5640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:29.395198 2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26c50a93eb842b183f5753c69b6795c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:29.395201 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26c50a93eb842b183f5753c69b6795c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:29.395224 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26c50a93eb842b183f5753c69b6795c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:29.396654 2707 tablet_bootstrap.cc:492] T b26c50a93eb842b183f5753c69b6795c P 4712dcc9dcc54da19007ae2c1832554c: Bootstrap starting.
I20260926 09:27:29.396689 2708 tablet_bootstrap.cc:492] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec: Bootstrap starting.
I20260926 09:27:29.397080 2708 tablet_bootstrap.cc:654] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:29.397092 2707 tablet_bootstrap.cc:654] T b26c50a93eb842b183f5753c69b6795c P 4712dcc9dcc54da19007ae2c1832554c: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:29.397568 2709 tablet_bootstrap.cc:492] T b26c50a93eb842b183f5753c69b6795c P eb0bc8efbdfd4537a1ae503077a60c4d: Bootstrap starting.
I20260926 09:27:29.397737 2707 tablet_bootstrap.cc:492] T b26c50a93eb842b183f5753c69b6795c P 4712dcc9dcc54da19007ae2c1832554c: No bootstrap required, opened a new log
I20260926 09:27:29.397794 2707 ts_tablet_manager.cc:1397] T b26c50a93eb842b183f5753c69b6795c P 4712dcc9dcc54da19007ae2c1832554c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:29.397914 2708 tablet_bootstrap.cc:492] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec: No bootstrap required, opened a new log
I20260926 09:27:29.397919 2707 raft_consensus.cc:348] T b26c50a93eb842b183f5753c69b6795c P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:29.397969 2708 ts_tablet_manager.cc:1397] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:29.397995 2707 raft_consensus.cc:374] T b26c50a93eb842b183f5753c69b6795c P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:29.398013 2709 tablet_bootstrap.cc:654] T b26c50a93eb842b183f5753c69b6795c P eb0bc8efbdfd4537a1ae503077a60c4d: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:29.398041 2707 raft_consensus.cc:729] T b26c50a93eb842b183f5753c69b6795c P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712dcc9dcc54da19007ae2c1832554c, State: Initialized, Role: FOLLOWER
I20260926 09:27:29.398106 2707 consensus_queue.cc:260] T b26c50a93eb842b183f5753c69b6795c P 4712dcc9dcc54da19007ae2c1832554c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:29.398100 2708 raft_consensus.cc:348] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:29.398166 2708 raft_consensus.cc:374] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:29.398195 2708 raft_consensus.cc:729] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519583d12f9c4e7bac953975b7c7c6ec, State: Initialized, Role: FOLLOWER
I20260926 09:27:29.398221 2707 ts_tablet_manager.cc:1428] T b26c50a93eb842b183f5753c69b6795c P 4712dcc9dcc54da19007ae2c1832554c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:29.398259 2708 consensus_queue.cc:260] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:29.398432 2708 ts_tablet_manager.cc:1428] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:29.398989 2709 tablet_bootstrap.cc:492] T b26c50a93eb842b183f5753c69b6795c P eb0bc8efbdfd4537a1ae503077a60c4d: No bootstrap required, opened a new log
I20260926 09:27:29.399046 2709 ts_tablet_manager.cc:1397] T b26c50a93eb842b183f5753c69b6795c P eb0bc8efbdfd4537a1ae503077a60c4d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 09:27:29.399200 2709 raft_consensus.cc:348] T b26c50a93eb842b183f5753c69b6795c P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:29.399298 2709 raft_consensus.cc:374] T b26c50a93eb842b183f5753c69b6795c P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:29.399325 2709 raft_consensus.cc:729] T b26c50a93eb842b183f5753c69b6795c P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb0bc8efbdfd4537a1ae503077a60c4d, State: Initialized, Role: FOLLOWER
I20260926 09:27:29.399369 2709 consensus_queue.cc:260] T b26c50a93eb842b183f5753c69b6795c P eb0bc8efbdfd4537a1ae503077a60c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:29.399545 2709 ts_tablet_manager.cc:1428] T b26c50a93eb842b183f5753c69b6795c P eb0bc8efbdfd4537a1ae503077a60c4d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 09:27:29.621054 2712 raft_consensus.cc:482] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:27:29.621165 2712 raft_consensus.cc:504] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:29.621556 2712 leader_election.cc:288] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712dcc9dcc54da19007ae2c1832554c (127.0.98.2:36885), eb0bc8efbdfd4537a1ae503077a60c4d (127.0.98.3:34015)
I20260926 09:27:29.625712 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:29.621785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34015 (local address 127.0.98.1:41109)
0926 09:27:29.621970 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:29.622066 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:29.622125 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:29.622327 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:29.622328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:29.622329 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:29.622407 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:29.622409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:29.623436 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:29.623437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:29.624690 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:29.624706 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:29.625399 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:29.625400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:29.625471 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:29.625475 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:27:29.625545 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 09:27:29.625589 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":3,"thread_start_us":112,"threads_started":1}
I20260926 09:27:29.625705 2717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:29.621859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41109 (local address 127.0.98.3:34015)
0926 09:27:29.622018 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:29.622021 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:29.622169 (+ 148us) server_negotiation.cc:407] Connection header received
0926 09:27:29.622200 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:29.622201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:29.622212 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:29.622278 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:29.622521 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:29.623309 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:29.624784 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:29.625316 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:29.625399 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:29.625410 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:29.625570 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 09:27:29.625631 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
I20260926 09:27:29.625958 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:29.621785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36885 (local address 127.0.98.1:54523)
0926 09:27:29.621970 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:29.621981 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:29.622040 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:29.622307 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:29.622308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:29.622311 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:29.622391 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:29.622393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:29.623401 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:29.623401 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:29.625027 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:29.625030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:29.625666 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:29.625667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:29.625739 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:29.625744 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:27:29.625820 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 09:27:29.625873 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":116,"threads_started":1}
I20260926 09:27:29.625952 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26c50a93eb842b183f5753c69b6795c" candidate_uuid: "519583d12f9c4e7bac953975b7c7c6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" is_pre_election: true
I20260926 09:27:29.626039 2556 raft_consensus.cc:2454] T b26c50a93eb842b183f5753c69b6795c P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 519583d12f9c4e7bac953975b7c7c6ec in term 0.
I20260926 09:27:29.626081 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:29.621862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54523 (local address 127.0.98.2:36885)
0926 09:27:29.622020 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:29.622022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:29.622033 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:27:29.622177 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:29.622178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:29.622189 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:29.622300 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:29.622517 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:29.623310 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:29.625154 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:29.625532 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:29.625660 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:29.625671 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:29.625943 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 09:27:29.625996 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
I20260926 09:27:29.626325 2207 leader_election.cc:302] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519583d12f9c4e7bac953975b7c7c6ec, eb0bc8efbdfd4537a1ae503077a60c4d; no voters:
I20260926 09:27:29.626302 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26c50a93eb842b183f5753c69b6795c" candidate_uuid: "519583d12f9c4e7bac953975b7c7c6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712dcc9dcc54da19007ae2c1832554c" is_pre_election: true
I20260926 09:27:29.626444 2416 raft_consensus.cc:2454] T b26c50a93eb842b183f5753c69b6795c P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 519583d12f9c4e7bac953975b7c7c6ec in term 0.
I20260926 09:27:29.626469 2712 raft_consensus.cc:2781] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:27:29.626523 2712 raft_consensus.cc:482] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:27:29.626544 2712 raft_consensus.cc:3037] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:29.628509 2712 raft_consensus.cc:504] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:29.628674 2712 leader_election.cc:288] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec [CANDIDATE]: Term 1 election: Requested vote from peers 4712dcc9dcc54da19007ae2c1832554c (127.0.98.2:36885), eb0bc8efbdfd4537a1ae503077a60c4d (127.0.98.3:34015)
I20260926 09:27:29.628811 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26c50a93eb842b183f5753c69b6795c" candidate_uuid: "519583d12f9c4e7bac953975b7c7c6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d"
I20260926 09:27:29.628894 2556 raft_consensus.cc:3037] T b26c50a93eb842b183f5753c69b6795c P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:29.628916 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26c50a93eb842b183f5753c69b6795c" candidate_uuid: "519583d12f9c4e7bac953975b7c7c6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712dcc9dcc54da19007ae2c1832554c"
I20260926 09:27:29.629001 2416 raft_consensus.cc:3037] T b26c50a93eb842b183f5753c69b6795c P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:29.629683 2416 raft_consensus.cc:2454] T b26c50a93eb842b183f5753c69b6795c P 4712dcc9dcc54da19007ae2c1832554c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 519583d12f9c4e7bac953975b7c7c6ec in term 1.
I20260926 09:27:29.629758 2556 raft_consensus.cc:2454] T b26c50a93eb842b183f5753c69b6795c P eb0bc8efbdfd4537a1ae503077a60c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 519583d12f9c4e7bac953975b7c7c6ec in term 1.
I20260926 09:27:29.629879 2210 leader_election.cc:302] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712dcc9dcc54da19007ae2c1832554c, 519583d12f9c4e7bac953975b7c7c6ec; no voters:
I20260926 09:27:29.629976 2712 raft_consensus.cc:2781] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:29.630131 2712 raft_consensus.cc:686] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec [term 1 LEADER]: Becoming Leader. State: Replica: 519583d12f9c4e7bac953975b7c7c6ec, State: Running, Role: LEADER
I20260926 09:27:29.630221 2712 consensus_queue.cc:237] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:29.631026 2119 catalog_manager.cc:5495] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 519583d12f9c4e7bac953975b7c7c6ec (127.0.98.1). New cstate: current_term: 1 leader_uuid: "519583d12f9c4e7bac953975b7c7c6ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } health_report { overall_health: UNKNOWN } } }
I20260926 09:27:29.685591 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:29.682906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.0.98.1:34803)
0926 09:27:29.683105 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:29.683107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:29.683128 (+ 21us) server_negotiation.cc:407] Connection header received
0926 09:27:29.683163 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:29.683163 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:29.683184 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:29.683255 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:27:29.683400 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:29.684012 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:29.684786 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:29.685008 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:29.685073 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:29.685212 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:27:29.685316 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:27:29.685369 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:29.685409 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 09:27:29.685439 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":152,"threads_started":1}
I20260926 09:27:29.690551 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:29.688092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43683 (local address 127.0.98.2:36885)
0926 09:27:29.688171 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:29.688173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:29.688273 (+ 100us) server_negotiation.cc:407] Connection header received
0926 09:27:29.688338 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:29.688339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:29.688352 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:29.688402 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:27:29.688530 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:29.689040 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:29.689810 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:29.690057 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:29.690128 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:29.690314 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:27:29.690346 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:27:29.690390 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:29.690455 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 09:27:29.690474 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 09:27:29.690579 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:29.688004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36885 (local address 127.0.98.1:43683)
0926 09:27:29.688196 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:29.688206 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:29.688284 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:29.688403 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:29.688403 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:29.688405 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:27:29.688463 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:29.688465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:29.689116 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:29.689118 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:29.689723 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:29.689725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:29.690151 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:29.690152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:29.690220 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:29.690232 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:27:29.690400 (+ 168us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:27:29.690402 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:27:29.690440 (+ 38us) client_negotiation.cc:240] Negotiation successful
0926 09:27:29.690486 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":96,"threads_started":1}
I20260926 09:27:29.692891 2556 raft_consensus.cc:1261] T b26c50a93eb842b183f5753c69b6795c P eb0bc8efbdfd4537a1ae503077a60c4d [term 1 FOLLOWER]: Refusing update from remote peer 519583d12f9c4e7bac953975b7c7c6ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:29.692970 2416 raft_consensus.cc:1261] T b26c50a93eb842b183f5753c69b6795c P 4712dcc9dcc54da19007ae2c1832554c [term 1 FOLLOWER]: Refusing update from remote peer 519583d12f9c4e7bac953975b7c7c6ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:29.693177 2712 consensus_queue.cc:1035] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:29.693329 2719 consensus_queue.cc:1035] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:29.700980 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:29.698558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37347 (local address 127.0.98.1:34803)
0926 09:27:29.698600 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:29.698602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:29.698730 (+ 128us) server_negotiation.cc:407] Connection header received
0926 09:27:29.698803 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:29.698804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:29.698814 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:29.698871 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:27:29.698991 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:29.699448 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:29.700332 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:29.700627 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:29.700721 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:29.700816 (+ 95us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:27:29.700841 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:27:29.700876 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:29.700906 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 09:27:29.700929 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:27:29.700999 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:29.698515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34803 (local address 127.0.98.2:37347)
0926 09:27:29.698643 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:29.698653 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:29.698744 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:29.698875 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:29.698876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:29.698877 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:27:29.698929 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:29.698930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:29.699532 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:29.699533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:29.700259 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:29.700262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:29.700718 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:29.700719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:29.700761 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:29.700765 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:27:29.700875 (+ 110us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:27:29.700876 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 09:27:29.700905 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 09:27:29.700937 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":85,"threads_started":1}
I20260926 09:27:29.707244 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:29.707356 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:29.707969 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f: Sending DeleteTablet for 3 replicas of tablet b26c50a93eb842b183f5753c69b6795c
I20260926 09:27:29.708194 2396 tablet_service.cc:1514] Processing DeleteTablet for tablet b26c50a93eb842b183f5753c69b6795c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50134
I20260926 09:27:29.708310 2536 tablet_service.cc:1514] Processing DeleteTablet for tablet b26c50a93eb842b183f5753c69b6795c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58684
I20260926 09:27:29.708330 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet b26c50a93eb842b183f5753c69b6795c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53432
I20260926 09:27:29.708405 2696 tablet_replica.cc:330] T b26c50a93eb842b183f5753c69b6795c P 4712dcc9dcc54da19007ae2c1832554c: stopping tablet replica
I20260926 09:27:29.708461 2696 raft_consensus.cc:2229] T b26c50a93eb842b183f5753c69b6795c P 4712dcc9dcc54da19007ae2c1832554c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:29.708492 2696 raft_consensus.cc:2258] T b26c50a93eb842b183f5753c69b6795c P 4712dcc9dcc54da19007ae2c1832554c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:29.708563 2697 tablet_replica.cc:330] T b26c50a93eb842b183f5753c69b6795c P eb0bc8efbdfd4537a1ae503077a60c4d: stopping tablet replica
I20260926 09:27:29.708617 2697 raft_consensus.cc:2229] T b26c50a93eb842b183f5753c69b6795c P eb0bc8efbdfd4537a1ae503077a60c4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:29.708552 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 09:27:29.708658 2697 raft_consensus.cc:2258] T b26c50a93eb842b183f5753c69b6795c P eb0bc8efbdfd4537a1ae503077a60c4d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 09:27:29.708711 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:27:29.708721 2698 tablet_replica.cc:330] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec: stopping tablet replica
I20260926 09:27:29.708770 2698 raft_consensus.cc:2229] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:27:29.708853 2698 raft_consensus.cc:2258] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:29.709327 2696 ts_tablet_manager.cc:1905] T b26c50a93eb842b183f5753c69b6795c P 4712dcc9dcc54da19007ae2c1832554c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:29.709343 2697 ts_tablet_manager.cc:1905] T b26c50a93eb842b183f5753c69b6795c P eb0bc8efbdfd4537a1ae503077a60c4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:29.710202 2536 tablet_service.cc:1467] Processing CreateTablet for tablet 5f775297299044208f6894c13a16c633 (DEFAULT_TABLE table=test-table [id=8cb5de5782714d18a1eee15b860388d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:29.710253 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 5f775297299044208f6894c13a16c633 (DEFAULT_TABLE table=test-table [id=8cb5de5782714d18a1eee15b860388d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:29.710363 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f775297299044208f6894c13a16c633. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:29.710479 2396 tablet_service.cc:1467] Processing CreateTablet for tablet 5f775297299044208f6894c13a16c633 (DEFAULT_TABLE table=test-table [id=8cb5de5782714d18a1eee15b860388d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:29.710530 2698 ts_tablet_manager.cc:1905] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:29.711557 2708 tablet_bootstrap.cc:492] T 5f775297299044208f6894c13a16c633 P 519583d12f9c4e7bac953975b7c7c6ec: Bootstrap starting.
I20260926 09:27:29.711931 2708 tablet_bootstrap.cc:654] T 5f775297299044208f6894c13a16c633 P 519583d12f9c4e7bac953975b7c7c6ec: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:29.712522 2708 tablet_bootstrap.cc:492] T 5f775297299044208f6894c13a16c633 P 519583d12f9c4e7bac953975b7c7c6ec: No bootstrap required, opened a new log
I20260926 09:27:29.712576 2708 ts_tablet_manager.cc:1397] T 5f775297299044208f6894c13a16c633 P 519583d12f9c4e7bac953975b7c7c6ec: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:29.712735 2708 raft_consensus.cc:348] T 5f775297299044208f6894c13a16c633 P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:29.712808 2708 raft_consensus.cc:374] T 5f775297299044208f6894c13a16c633 P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:29.712833 2708 raft_consensus.cc:729] T 5f775297299044208f6894c13a16c633 P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519583d12f9c4e7bac953975b7c7c6ec, State: Initialized, Role: FOLLOWER
I20260926 09:27:29.712880 2708 consensus_queue.cc:260] T 5f775297299044208f6894c13a16c633 P 519583d12f9c4e7bac953975b7c7c6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:29.712996 2708 ts_tablet_manager.cc:1428] T 5f775297299044208f6894c13a16c633 P 519583d12f9c4e7bac953975b7c7c6ec: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:29.721225 2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f775297299044208f6894c13a16c633. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:29.721239 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f775297299044208f6894c13a16c633. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:29.722666 2709 tablet_bootstrap.cc:492] T 5f775297299044208f6894c13a16c633 P eb0bc8efbdfd4537a1ae503077a60c4d: Bootstrap starting.
I20260926 09:27:29.723250 2709 tablet_bootstrap.cc:654] T 5f775297299044208f6894c13a16c633 P eb0bc8efbdfd4537a1ae503077a60c4d: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:29.723832 2696 ts_tablet_manager.cc:1918] T b26c50a93eb842b183f5753c69b6795c P 4712dcc9dcc54da19007ae2c1832554c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:27:29.723892 2696 log.cc:1196] T b26c50a93eb842b183f5753c69b6795c P 4712dcc9dcc54da19007ae2c1832554c: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-1/wal/wals/b26c50a93eb842b183f5753c69b6795c
I20260926 09:27:29.724025 2707 tablet_bootstrap.cc:492] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c: Bootstrap starting.
I20260926 09:27:29.724112 2696 ts_tablet_manager.cc:1939] T b26c50a93eb842b183f5753c69b6795c P 4712dcc9dcc54da19007ae2c1832554c: Deleting consensus metadata
I20260926 09:27:29.724313 2707 tablet_bootstrap.cc:654] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:29.725484 2707 tablet_bootstrap.cc:492] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c: No bootstrap required, opened a new log
I20260926 09:27:29.725541 2707 ts_tablet_manager.cc:1397] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 09:27:29.725695 2707 raft_consensus.cc:348] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:29.725783 2707 raft_consensus.cc:374] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:29.725807 2707 raft_consensus.cc:729] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712dcc9dcc54da19007ae2c1832554c, State: Initialized, Role: FOLLOWER
I20260926 09:27:29.725853 2707 consensus_queue.cc:260] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:29.725992 2707 ts_tablet_manager.cc:1428] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:29.726097 2709 tablet_bootstrap.cc:492] T 5f775297299044208f6894c13a16c633 P eb0bc8efbdfd4537a1ae503077a60c4d: No bootstrap required, opened a new log
I20260926 09:27:29.726145 2709 ts_tablet_manager.cc:1397] T 5f775297299044208f6894c13a16c633 P eb0bc8efbdfd4537a1ae503077a60c4d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260926 09:27:29.726293 2709 raft_consensus.cc:348] T 5f775297299044208f6894c13a16c633 P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:29.726354 2709 raft_consensus.cc:374] T 5f775297299044208f6894c13a16c633 P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:29.726372 2709 raft_consensus.cc:729] T 5f775297299044208f6894c13a16c633 P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb0bc8efbdfd4537a1ae503077a60c4d, State: Initialized, Role: FOLLOWER
I20260926 09:27:29.726441 2709 consensus_queue.cc:260] T 5f775297299044208f6894c13a16c633 P eb0bc8efbdfd4537a1ae503077a60c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:29.726539 2709 ts_tablet_manager.cc:1428] T 5f775297299044208f6894c13a16c633 P eb0bc8efbdfd4537a1ae503077a60c4d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:29.726913 2697 ts_tablet_manager.cc:1918] T b26c50a93eb842b183f5753c69b6795c P eb0bc8efbdfd4537a1ae503077a60c4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:27:29.727021 2697 log.cc:1196] T b26c50a93eb842b183f5753c69b6795c P eb0bc8efbdfd4537a1ae503077a60c4d: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/b26c50a93eb842b183f5753c69b6795c
I20260926 09:27:29.727234 2697 ts_tablet_manager.cc:1939] T b26c50a93eb842b183f5753c69b6795c P eb0bc8efbdfd4537a1ae503077a60c4d: Deleting consensus metadata
I20260926 09:27:29.745414 2106 catalog_manager.cc:4841] TS 4712dcc9dcc54da19007ae2c1832554c (127.0.98.2:36885): tablet b26c50a93eb842b183f5753c69b6795c (table test-table [id=fab73127c4474504af960a7775dc5640]) successfully deleted
I20260926 09:27:29.749658 2636 raft_consensus.cc:482] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:27:29.749750 2636 raft_consensus.cc:504] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:29.749900 2636 leader_election.cc:288] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 519583d12f9c4e7bac953975b7c7c6ec (127.0.98.1:34803), eb0bc8efbdfd4537a1ae503077a60c4d (127.0.98.3:34015)
I20260926 09:27:29.750120 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f775297299044208f6894c13a16c633" candidate_uuid: "4712dcc9dcc54da19007ae2c1832554c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" is_pre_election: true
I20260926 09:27:29.750237 2556 raft_consensus.cc:2454] T 5f775297299044208f6894c13a16c633 P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712dcc9dcc54da19007ae2c1832554c in term 0.
I20260926 09:27:29.750211 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f775297299044208f6894c13a16c633" candidate_uuid: "4712dcc9dcc54da19007ae2c1832554c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519583d12f9c4e7bac953975b7c7c6ec" is_pre_election: true
I20260926 09:27:29.750291 2276 raft_consensus.cc:2454] T 5f775297299044208f6894c13a16c633 P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712dcc9dcc54da19007ae2c1832554c in term 0.
I20260926 09:27:29.750399 2347 leader_election.cc:302] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712dcc9dcc54da19007ae2c1832554c, eb0bc8efbdfd4537a1ae503077a60c4d; no voters:
I20260926 09:27:29.750515 2636 raft_consensus.cc:2781] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:27:29.750571 2636 raft_consensus.cc:482] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:27:29.750592 2636 raft_consensus.cc:3037] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:29.753840 2698 ts_tablet_manager.cc:1918] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:27:29.753921 2698 log.cc:1196] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-0/wal/wals/b26c50a93eb842b183f5753c69b6795c
I20260926 09:27:29.754158 2698 ts_tablet_manager.cc:1939] T b26c50a93eb842b183f5753c69b6795c P 519583d12f9c4e7bac953975b7c7c6ec: Deleting consensus metadata
I20260926 09:27:29.754891 2103 catalog_manager.cc:4841] TS 519583d12f9c4e7bac953975b7c7c6ec (127.0.98.1:34803): tablet b26c50a93eb842b183f5753c69b6795c (table test-table [id=fab73127c4474504af960a7775dc5640]) successfully deleted
I20260926 09:27:29.788568 2103 catalog_manager.cc:4841] TS eb0bc8efbdfd4537a1ae503077a60c4d (127.0.98.3:34015): tablet b26c50a93eb842b183f5753c69b6795c (table test-table [id=fab73127c4474504af960a7775dc5640]) successfully deleted
I20260926 09:27:29.803612 2636 raft_consensus.cc:504] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:29.803839 2636 leader_election.cc:288] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c [CANDIDATE]: Term 1 election: Requested vote from peers 519583d12f9c4e7bac953975b7c7c6ec (127.0.98.1:34803), eb0bc8efbdfd4537a1ae503077a60c4d (127.0.98.3:34015)
I20260926 09:27:29.804138 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f775297299044208f6894c13a16c633" candidate_uuid: "4712dcc9dcc54da19007ae2c1832554c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519583d12f9c4e7bac953975b7c7c6ec"
I20260926 09:27:29.804167 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f775297299044208f6894c13a16c633" candidate_uuid: "4712dcc9dcc54da19007ae2c1832554c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d"
I20260926 09:27:29.804242 2276 raft_consensus.cc:3037] T 5f775297299044208f6894c13a16c633 P 519583d12f9c4e7bac953975b7c7c6ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:29.804253 2556 raft_consensus.cc:3037] T 5f775297299044208f6894c13a16c633 P eb0bc8efbdfd4537a1ae503077a60c4d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:29.805236 2556 raft_consensus.cc:2454] T 5f775297299044208f6894c13a16c633 P eb0bc8efbdfd4537a1ae503077a60c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712dcc9dcc54da19007ae2c1832554c in term 1.
I20260926 09:27:29.805434 2276 raft_consensus.cc:2454] T 5f775297299044208f6894c13a16c633 P 519583d12f9c4e7bac953975b7c7c6ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712dcc9dcc54da19007ae2c1832554c in term 1.
I20260926 09:27:29.805843 2347 leader_election.cc:302] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712dcc9dcc54da19007ae2c1832554c, 519583d12f9c4e7bac953975b7c7c6ec; no voters:
I20260926 09:27:29.806047 2636 raft_consensus.cc:2781] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:29.806099 2636 raft_consensus.cc:686] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c [term 1 LEADER]: Becoming Leader. State: Replica: 4712dcc9dcc54da19007ae2c1832554c, State: Running, Role: LEADER
I20260926 09:27:29.806152 2636 consensus_queue.cc:237] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } }
I20260926 09:27:29.806730 2119 catalog_manager.cc:5495] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4712dcc9dcc54da19007ae2c1832554c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4712dcc9dcc54da19007ae2c1832554c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712dcc9dcc54da19007ae2c1832554c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 } health_report { overall_health: UNKNOWN } } }
I20260926 09:27:29.835561 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:29.833320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50156 (local address 127.0.98.2:36885)
0926 09:27:29.833368 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:29.833370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:29.833379 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:27:29.833439 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:29.833440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:29.833451 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:29.833500 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:27:29.833652 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:29.834131 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:29.834872 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:29.835087 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:29.835166 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:29.835293 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:27:29.835328 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:27:29.835395 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:29.835452 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 09:27:29.835473 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:27:29.838121 2276 raft_consensus.cc:1261] T 5f775297299044208f6894c13a16c633 P 519583d12f9c4e7bac953975b7c7c6ec [term 1 FOLLOWER]: Refusing update from remote peer 4712dcc9dcc54da19007ae2c1832554c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:29.838081 2556 raft_consensus.cc:1261] T 5f775297299044208f6894c13a16c633 P eb0bc8efbdfd4537a1ae503077a60c4d [term 1 FOLLOWER]: Refusing update from remote peer 4712dcc9dcc54da19007ae2c1832554c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:29.838340 2636 consensus_queue.cc:1035] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0bc8efbdfd4537a1ae503077a60c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:29.838469 2684 consensus_queue.cc:1035] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c [LEADER]: Connected to new peer: Peer: permanent_uuid: "519583d12f9c4e7bac953975b7c7c6ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 09:27:29.842715 2677 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 09:27:31.360227 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:31.360347 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:31.361155 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c8cf53869c446919b7d2de4560aeb2f: Sending DeleteTablet for 3 replicas of tablet 5f775297299044208f6894c13a16c633
I20260926 09:27:31.361426 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f775297299044208f6894c13a16c633 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50134
I20260926 09:27:31.361474 2536 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f775297299044208f6894c13a16c633 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58684
I20260926 09:27:31.361475 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f775297299044208f6894c13a16c633 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53432
I20260926 09:27:31.361727 2762 tablet_replica.cc:330] T 5f775297299044208f6894c13a16c633 P eb0bc8efbdfd4537a1ae503077a60c4d: stopping tablet replica
I20260926 09:27:31.361783 2762 raft_consensus.cc:2229] T 5f775297299044208f6894c13a16c633 P eb0bc8efbdfd4537a1ae503077a60c4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:31.361822 2762 raft_consensus.cc:2258] T 5f775297299044208f6894c13a16c633 P eb0bc8efbdfd4537a1ae503077a60c4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:31.361855 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 2191
I20260926 09:27:31.363479 2762 ts_tablet_manager.cc:1905] T 5f775297299044208f6894c13a16c633 P eb0bc8efbdfd4537a1ae503077a60c4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:31.364552 2760 tablet_replica.cc:330] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c: stopping tablet replica
I20260926 09:27:31.364629 2760 raft_consensus.cc:2229] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:27:31.364694 2760 raft_consensus.cc:2258] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:31.364926 2760 ts_tablet_manager.cc:1905] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:31.365399 2762 ts_tablet_manager.cc:1918] T 5f775297299044208f6894c13a16c633 P eb0bc8efbdfd4537a1ae503077a60c4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 09:27:31.365465 2762 log.cc:1196] T 5f775297299044208f6894c13a16c633 P eb0bc8efbdfd4537a1ae503077a60c4d: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/5f775297299044208f6894c13a16c633
I20260926 09:27:31.365717 2762 ts_tablet_manager.cc:1939] T 5f775297299044208f6894c13a16c633 P eb0bc8efbdfd4537a1ae503077a60c4d: Deleting consensus metadata
I20260926 09:27:31.366130 2103 catalog_manager.cc:4841] TS eb0bc8efbdfd4537a1ae503077a60c4d (127.0.98.3:34015): tablet 5f775297299044208f6894c13a16c633 (table test-table [id=8cb5de5782714d18a1eee15b860388d4]) successfully deleted
I20260926 09:27:31.366361 2760 ts_tablet_manager.cc:1918] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 09:27:31.366432 2760 log.cc:1196] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414839069479-392-0/minicluster-data/ts-1/wal/wals/5f775297299044208f6894c13a16c633
I20260926 09:27:31.366667 2760 ts_tablet_manager.cc:1939] T 5f775297299044208f6894c13a16c633 P 4712dcc9dcc54da19007ae2c1832554c: Deleting consensus metadata
I20260926 09:27:31.366986 2106 catalog_manager.cc:4841] TS 4712dcc9dcc54da19007ae2c1832554c (127.0.98.2:36885): tablet 5f775297299044208f6894c13a16c633 (table test-table [id=8cb5de5782714d18a1eee15b860388d4]) successfully deleted
W20260926 09:27:31.369477 2103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34803 (error 108)
I20260926 09:27:31.369733 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 2331
I20260926 09:27:31.374130 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:31.373907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34803 (local address 127.0.0.1:53446)
0926 09:27:31.374050 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:31.374092 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
W20260926 09:27:31.374286 2103 catalog_manager.cc:4584] TS 519583d12f9c4e7bac953975b7c7c6ec (127.0.98.1:34803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f775297299044208f6894c13a16c633: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34803: connect: Connection refused (error 111)
I20260926 09:27:31.376056 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 2471
I20260926 09:27:31.381068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 2087
2026-09-26T09:27:31Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5886 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:27:31 dist-test-slave-dfzm krb5kdc[2767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:27:31 dist-test-slave-dfzm krb5kdc[2767](info): set up 2 sockets
Sep 26 09:27:31 dist-test-slave-dfzm krb5kdc[2767](info): commencing operation
krb5kdc: starting...
W20260926 09:27:32.437338 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:27:32 dist-test-slave-dfzm krb5kdc[2767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:27:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:27:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:27:32.552702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46153
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:32.633044 2780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:32.633258 2780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:32.633280 2780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:32.634486 2780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:27:32.634531 2780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:32.634545 2780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:32.634558 2780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:27:32.634608 2780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:27:32.636111 2780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:32.636361 2780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:32.636638 2780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:32.641408 2786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:32.641470 2788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:32.641470 2785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:32.641662 2780 server_base.cc:1034] running on GCE node
I20260926 09:27:32.641928 2780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:32.642172 2780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:32.643322 2780 hybrid_clock.cc:648] HybridClock initialized: now 1790414852643306 us; error 41 us; skew 500 ppm
Sep 26 09:27:32 dist-test-slave-dfzm krb5kdc[2767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:32.645779 2780 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:27:32.646170 2780 webserver.cc:460] Webserver started at http://127.0.98.62:44113/ using document root <none> and password file <none>
I20260926 09:27:32.646365 2780 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:32.646406 2780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:32.646502 2780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:32.647571 2780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "70c3de4b2ebf4e57b353bae0c5d89ff8"
format_stamp: "Formatted at 2026-09-26 09:27:32 on dist-test-slave-dfzm"
server_key: "b5a83a8eb79b3a279c1f63fe034cc363"
server_key_iv: "f1a53ffcddf87cd71069de6d80f725f7"
server_key_version: "encryptionkey@0"
I20260926 09:27:32.647866 2780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "70c3de4b2ebf4e57b353bae0c5d89ff8"
format_stamp: "Formatted at 2026-09-26 09:27:32 on dist-test-slave-dfzm"
server_key: "b5a83a8eb79b3a279c1f63fe034cc363"
server_key_iv: "f1a53ffcddf87cd71069de6d80f725f7"
server_key_version: "encryptionkey@0"
I20260926 09:27:32.649195 2780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:27:32.649946 2795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:32.650106 2780 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:27:32.650192 2780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "70c3de4b2ebf4e57b353bae0c5d89ff8"
format_stamp: "Formatted at 2026-09-26 09:27:32 on dist-test-slave-dfzm"
server_key: "b5a83a8eb79b3a279c1f63fe034cc363"
server_key_iv: "f1a53ffcddf87cd71069de6d80f725f7"
server_key_version: "encryptionkey@0"
I20260926 09:27:32.650249 2780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:32.660341 2780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:32.662283 2780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:32.662395 2780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:32.666100 2780 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46153
I20260926 09:27:32.666373 2847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46153 every 8 connection(s)
I20260926 09:27:32.666463 2780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:27:32.667016 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 2780
I20260926 09:27:32.667091 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:27:32.667264 2848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:32.667286 392 external_mini_cluster.cc:1428] Setting key 9f8210a49db1100db63549d42966e949
I20260926 09:27:32.670590 2848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70c3de4b2ebf4e57b353bae0c5d89ff8: Bootstrap starting.
I20260926 09:27:32.671212 2848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70c3de4b2ebf4e57b353bae0c5d89ff8: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:32.671561 2848 log.cc:824] T 00000000000000000000000000000000 P 70c3de4b2ebf4e57b353bae0c5d89ff8: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:32.672317 2848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70c3de4b2ebf4e57b353bae0c5d89ff8: No bootstrap required, opened a new log
Sep 26 09:27:32 dist-test-slave-dfzm krb5kdc[2767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:32.674580 2848 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70c3de4b2ebf4e57b353bae0c5d89ff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c3de4b2ebf4e57b353bae0c5d89ff8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46153 } }
I20260926 09:27:32.674751 2848 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70c3de4b2ebf4e57b353bae0c5d89ff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:32.674778 2848 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70c3de4b2ebf4e57b353bae0c5d89ff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c3de4b2ebf4e57b353bae0c5d89ff8, State: Initialized, Role: FOLLOWER
I20260926 09:27:32.674924 2848 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70c3de4b2ebf4e57b353bae0c5d89ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c3de4b2ebf4e57b353bae0c5d89ff8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46153 } }
I20260926 09:27:32.675012 2848 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70c3de4b2ebf4e57b353bae0c5d89ff8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:27:32.675045 2848 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70c3de4b2ebf4e57b353bae0c5d89ff8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:27:32.675099 2848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70c3de4b2ebf4e57b353bae0c5d89ff8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:32.675706 2848 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70c3de4b2ebf4e57b353bae0c5d89ff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c3de4b2ebf4e57b353bae0c5d89ff8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46153 } }
I20260926 09:27:32.675837 2848 leader_election.cc:302] T 00000000000000000000000000000000 P 70c3de4b2ebf4e57b353bae0c5d89ff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70c3de4b2ebf4e57b353bae0c5d89ff8; no voters:
I20260926 09:27:32.675998 2848 leader_election.cc:288] T 00000000000000000000000000000000 P 70c3de4b2ebf4e57b353bae0c5d89ff8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:27:32.676034 2853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70c3de4b2ebf4e57b353bae0c5d89ff8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:32.676143 2853 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70c3de4b2ebf4e57b353bae0c5d89ff8 [term 1 LEADER]: Becoming Leader. State: Replica: 70c3de4b2ebf4e57b353bae0c5d89ff8, State: Running, Role: LEADER
I20260926 09:27:32.676257 2848 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70c3de4b2ebf4e57b353bae0c5d89ff8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:27:32.676291 2853 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70c3de4b2ebf4e57b353bae0c5d89ff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c3de4b2ebf4e57b353bae0c5d89ff8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46153 } }
I20260926 09:27:32.676648 2853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70c3de4b2ebf4e57b353bae0c5d89ff8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70c3de4b2ebf4e57b353bae0c5d89ff8. Latest consensus state: current_term: 1 leader_uuid: "70c3de4b2ebf4e57b353bae0c5d89ff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c3de4b2ebf4e57b353bae0c5d89ff8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46153 } } }
I20260926 09:27:32.676661 2855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70c3de4b2ebf4e57b353bae0c5d89ff8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70c3de4b2ebf4e57b353bae0c5d89ff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c3de4b2ebf4e57b353bae0c5d89ff8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46153 } } }
I20260926 09:27:32.676851 2855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70c3de4b2ebf4e57b353bae0c5d89ff8 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:32.677070 2853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70c3de4b2ebf4e57b353bae0c5d89ff8 [sys.catalog]: This master's current role is: LEADER
W20260926 09:27:32.678143 2869 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70c3de4b2ebf4e57b353bae0c5d89ff8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:27:32.678201 2869 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:27:32.678258 2861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:27:32.678444 2861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:27:32.678709 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:32.667771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.0.98.62:46153)
0926 09:27:32.667954 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:32.667964 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:32.668007 (+ 43us) server_negotiation.cc:407] Connection header received
0926 09:27:32.668423 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:32.668432 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:32.668493 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:32.668656 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:32.669240 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:32.670669 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:32.671427 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:32.671986 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:32.672105 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:32.674658 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:32.674662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:32.674663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:32.674671 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:32.676726 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:32.677174 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:32.677178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:32.677178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:32.677285 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:32.677503 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:32.677503 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:32.677504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:32.677827 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:32.678416 (+ 589us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:32.678474 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 09:27:32.678572 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260926 09:27:32.679903 2861 catalog_manager.cc:1347] Generated new cluster ID: 5e5f90400c22484abd1a9c29313b64d5
I20260926 09:27:32.679965 2861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:27:32.685431 2861 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:27:32.686249 2861 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:27:32.698660 2861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70c3de4b2ebf4e57b353bae0c5d89ff8: Generated new TSK 0
I20260926 09:27:32.698926 2861 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:27:32.728834 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46153
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:32.810063 2874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:32.810278 2874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:32.810302 2874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:32.811483 2874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:32.811528 2874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:32.811580 2874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:27:32.813230 2874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:32.813496 2874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:32.813759 2874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:32.818717 2882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:32.818693 2880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:32.818696 2879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:32.818979 2874 server_base.cc:1034] running on GCE node
I20260926 09:27:32.819242 2874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:32.819455 2874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:32.820600 2874 hybrid_clock.cc:648] HybridClock initialized: now 1790414852820577 us; error 38 us; skew 500 ppm
Sep 26 09:27:32 dist-test-slave-dfzm krb5kdc[2767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:32.823203 2874 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:27:32.823679 2874 webserver.cc:460] Webserver started at http://127.0.98.1:43529/ using document root <none> and password file <none>
I20260926 09:27:32.823925 2874 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:32.823995 2874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:32.824148 2874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:32.825238 2874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "00662f6dc31241eb82d59fe6d5c67319"
format_stamp: "Formatted at 2026-09-26 09:27:32 on dist-test-slave-dfzm"
server_key: "8d21a54e6f5cd33a2a232f6af20c0a88"
server_key_iv: "d9c9989a4e8ee8f9bea5c2e160a43399"
server_key_version: "encryptionkey@0"
I20260926 09:27:32.825662 2874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "00662f6dc31241eb82d59fe6d5c67319"
format_stamp: "Formatted at 2026-09-26 09:27:32 on dist-test-slave-dfzm"
server_key: "8d21a54e6f5cd33a2a232f6af20c0a88"
server_key_iv: "d9c9989a4e8ee8f9bea5c2e160a43399"
server_key_version: "encryptionkey@0"
I20260926 09:27:32.827008 2874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:27:32.827847 2889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:32.828145 2874 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:27:32.828248 2874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "00662f6dc31241eb82d59fe6d5c67319"
format_stamp: "Formatted at 2026-09-26 09:27:32 on dist-test-slave-dfzm"
server_key: "8d21a54e6f5cd33a2a232f6af20c0a88"
server_key_iv: "d9c9989a4e8ee8f9bea5c2e160a43399"
server_key_version: "encryptionkey@0"
I20260926 09:27:32.828361 2874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:32.850477 2874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:32.852502 2874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:32.852672 2874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:32.852950 2874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:32.853271 2874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:32.853327 2874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:32.853367 2874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:32.853396 2874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:32.859259 2874 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45277
I20260926 09:27:32.859308 3002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45277 every 8 connection(s)
I20260926 09:27:32.859650 2874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:27:32.863998 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 2874
I20260926 09:27:32.864107 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:27:32.864277 392 external_mini_cluster.cc:1428] Setting key a70b8f644576f91000090540d82620a2
Sep 26 09:27:32 dist-test-slave-dfzm krb5kdc[2767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:32.869132 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:32.860391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55279 (local address 127.0.98.62:46153)
0926 09:27:32.860465 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:32.860467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:32.860697 (+ 230us) server_negotiation.cc:407] Connection header received
0926 09:27:32.861674 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:32.861680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:32.861691 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:32.861744 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:32.862367 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:32.862856 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:32.863838 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:32.864037 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:32.864122 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:32.866377 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:32.866378 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:32.866379 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:32.866386 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:32.867742 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:32.868151 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:32.868152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:32.868153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:32.868214 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:32.868482 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:32.868483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:32.868484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:32.868602 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:32.868678 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:32.868955 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 09:27:32.869041 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 09:27:32.869338 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:32.860371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46153 (local address 127.0.98.1:55279)
0926 09:27:32.860567 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:32.860602 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:32.861524 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:32.861807 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:32.861810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:32.862079 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:32.862279 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:32.862283 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:32.862936 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:32.862937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:32.863741 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:32.863743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:32.864131 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:32.864132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:32.864264 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:32.864771 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:32.864781 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:32.866276 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:32.867828 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:32.867834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:32.867837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:32.868070 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:32.868279 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:32.868280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:32.868280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:32.868397 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:32.868669 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:32.868675 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:32.868819 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 09:27:32.868929 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 09:27:32.869087 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":108,"threads_started":1}
I20260926 09:27:32.869779 3003 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46153
I20260926 09:27:32.869871 3003 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:32.870076 3003 heartbeater.cc:511] Master 127.0.98.62:46153 requested a full tablet report, sending...
I20260926 09:27:32.870579 2812 ts_manager.cc:194] Registered new tserver with Master: 00662f6dc31241eb82d59fe6d5c67319 (127.0.98.1:45277)
I20260926 09:27:32.871197 2812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55279
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:27:32.890583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46153
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:32.971462 3008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:32.971683 3008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:32.971706 3008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:32.972813 3008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:32.972849 3008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:32.972874 3008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:27:32.974326 3008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:32.974520 3008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:32.974735 3008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:32.979982 3013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:32.980029 3014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:32.980050 3016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:32.980154 3008 server_base.cc:1034] running on GCE node
I20260926 09:27:32.980420 3008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:32.980667 3008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:32.981812 3008 hybrid_clock.cc:648] HybridClock initialized: now 1790414852981799 us; error 29 us; skew 500 ppm
Sep 26 09:27:32 dist-test-slave-dfzm krb5kdc[2767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:32.984371 3008 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:27:32.984768 3008 webserver.cc:460] Webserver started at http://127.0.98.2:42233/ using document root <none> and password file <none>
I20260926 09:27:32.985009 3008 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:32.985055 3008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:32.985146 3008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:32.986224 3008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "aa3ee3389c984df8a73dbc980db83452"
format_stamp: "Formatted at 2026-09-26 09:27:32 on dist-test-slave-dfzm"
server_key: "603fb113629cfb72fe20dd3b662b04b8"
server_key_iv: "11db1abc0dd2117497120141a8bf1bf2"
server_key_version: "encryptionkey@0"
I20260926 09:27:32.986527 3008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "aa3ee3389c984df8a73dbc980db83452"
format_stamp: "Formatted at 2026-09-26 09:27:32 on dist-test-slave-dfzm"
server_key: "603fb113629cfb72fe20dd3b662b04b8"
server_key_iv: "11db1abc0dd2117497120141a8bf1bf2"
server_key_version: "encryptionkey@0"
I20260926 09:27:32.987798 3008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:27:32.988459 3023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:32.988636 3008 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:32.988726 3008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "aa3ee3389c984df8a73dbc980db83452"
format_stamp: "Formatted at 2026-09-26 09:27:32 on dist-test-slave-dfzm"
server_key: "603fb113629cfb72fe20dd3b662b04b8"
server_key_iv: "11db1abc0dd2117497120141a8bf1bf2"
server_key_version: "encryptionkey@0"
I20260926 09:27:32.988798 3008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:32.995932 3008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:32.997772 3008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:32.997931 3008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:32.998149 3008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:32.998445 3008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:32.998478 3008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:32.998500 3008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:32.998546 3008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:33.004328 3008 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45117
I20260926 09:27:33.004403 3136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45117 every 8 connection(s)
I20260926 09:27:33.004722 3008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:27:33.005306 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 3008
I20260926 09:27:33.005376 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:27:33.005618 392 external_mini_cluster.cc:1428] Setting key 4a159b3948b6d158d40af7114c012e92
Sep 26 09:27:33 dist-test-slave-dfzm krb5kdc[2767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 09:27:33.016057 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:33.005473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58183 (local address 127.0.98.62:46153)
0926 09:27:33.005528 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:33.005531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:33.005776 (+ 245us) server_negotiation.cc:407] Connection header received
0926 09:27:33.007070 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:33.007071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:33.007081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:33.007128 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:33.008045 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:33.008836 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:33.009973 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:33.010311 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:33.010391 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:33.012883 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:33.012885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:33.012885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:33.012894 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:33.014644 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:33.015050 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:33.015051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:33.015051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:33.015106 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:33.015381 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:33.015381 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:33.015382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:33.015516 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:33.015613 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:33.015909 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 09:27:33.015987 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:27:33.016263 3139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:33.005442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46153 (local address 127.0.98.2:58183)
0926 09:27:33.005633 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:33.005684 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:33.006947 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:33.007214 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:33.007218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:33.007625 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:33.007920 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:33.007926 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:33.008930 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:33.008931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:33.009861 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:33.009865 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:33.010407 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:33.010408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:33.010525 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:33.011130 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:33.011143 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:33.012776 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:33.014728 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:33.014735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:33.014737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:33.014964 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:33.015179 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:33.015181 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:33.015181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:33.015313 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:33.015620 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:33.015626 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:33.015785 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 09:27:33.015876 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 09:27:33.016006 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
I20260926 09:27:33.016806 3137 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46153
I20260926 09:27:33.016923 3137 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 09:27:33.017131 3137 heartbeater.cc:511] Master 127.0.98.62:46153 requested a full tablet report, sending...
I20260926 09:27:33.017668 2812 ts_manager.cc:194] Registered new tserver with Master: aa3ee3389c984df8a73dbc980db83452 (127.0.98.2:45117)
I20260926 09:27:33.018064 2812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:27:33.031847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46153
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:33.111898 3142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:33.112106 3142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:33.112129 3142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:33.113273 3142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:33.113314 3142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:33.113365 3142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:27:33.114883 3142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:33.115134 3142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:33.115355 3142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:33.120220 3148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:33.120225 3147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:33.120225 3150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:33.120580 3142 server_base.cc:1034] running on GCE node
I20260926 09:27:33.120823 3142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:33.121086 3142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:33.122263 3142 hybrid_clock.cc:648] HybridClock initialized: now 1790414853122251 us; error 47 us; skew 500 ppm
Sep 26 09:27:33 dist-test-slave-dfzm krb5kdc[2767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:33.124812 3142 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:27:33.125236 3142 webserver.cc:460] Webserver started at http://127.0.98.3:45911/ using document root <none> and password file <none>
I20260926 09:27:33.125455 3142 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:33.125530 3142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:33.125695 3142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:33.126778 3142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "f2f5ff3f9995402281f7673076721325"
format_stamp: "Formatted at 2026-09-26 09:27:33 on dist-test-slave-dfzm"
server_key: "a62d5782057e6a6b89ae604d1368e0ce"
server_key_iv: "9ef631e0b71a871978f09a50466c149c"
server_key_version: "encryptionkey@0"
I20260926 09:27:33.127130 3142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f2f5ff3f9995402281f7673076721325"
format_stamp: "Formatted at 2026-09-26 09:27:33 on dist-test-slave-dfzm"
server_key: "a62d5782057e6a6b89ae604d1368e0ce"
server_key_iv: "9ef631e0b71a871978f09a50466c149c"
server_key_version: "encryptionkey@0"
I20260926 09:27:33.128417 3142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:27:33.129089 3157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:33.129241 3142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:33.129345 3142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "f2f5ff3f9995402281f7673076721325"
format_stamp: "Formatted at 2026-09-26 09:27:33 on dist-test-slave-dfzm"
server_key: "a62d5782057e6a6b89ae604d1368e0ce"
server_key_iv: "9ef631e0b71a871978f09a50466c149c"
server_key_version: "encryptionkey@0"
I20260926 09:27:33.129418 3142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:33.146826 3142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:33.149305 3142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:33.149475 3142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:33.149747 3142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:33.150071 3142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:33.150126 3142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:33.150167 3142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:33.150213 3142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:33.155920 3142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39529
I20260926 09:27:33.155963 3270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39529 every 8 connection(s)
I20260926 09:27:33.156327 3142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:27:33.156678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 3142
I20260926 09:27:33.156780 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:27:33.156996 392 external_mini_cluster.cc:1428] Setting key 8c077da82f544041a3844a673942cae4
Sep 26 09:27:33 dist-test-slave-dfzm krb5kdc[2767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:33.165781 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:33.157128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59335 (local address 127.0.98.62:46153)
0926 09:27:33.157169 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:33.157171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:33.157382 (+ 211us) server_negotiation.cc:407] Connection header received
0926 09:27:33.158510 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:33.158511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:33.158519 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:33.158568 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:33.159135 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:33.159624 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:33.160580 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:33.160766 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:33.160832 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:33.162934 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:33.162936 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:33.162936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:33.162944 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:33.164398 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:33.164810 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:33.164811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:33.164812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:33.164872 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:33.165125 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:33.165126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:33.165126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:33.165238 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:33.165316 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:33.165610 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 09:27:33.165689 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 09:27:33.165977 3273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:33.157121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46153 (local address 127.0.98.3:59335)
0926 09:27:33.157279 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:33.157319 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:33.158412 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:33.158622 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:33.158626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:33.158868 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:33.159067 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:33.159071 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:33.159697 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:33.159698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:33.160485 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:33.160487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:33.160844 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:33.160845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:33.160938 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:33.161401 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:33.161409 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:33.162827 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:33.164499 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:33.164506 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:33.164508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:33.164731 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:33.164939 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:33.164940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:33.164940 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:33.165051 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:33.165309 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:33.165315 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:33.165455 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 09:27:33.165558 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 09:27:33.165728 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260926 09:27:33.166477 3271 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46153
I20260926 09:27:33.166556 3271 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:33.166733 3271 heartbeater.cc:511] Master 127.0.98.62:46153 requested a full tablet report, sending...
I20260926 09:27:33.167093 2812 ts_manager.cc:194] Registered new tserver with Master: f2f5ff3f9995402281f7673076721325 (127.0.98.3:39529)
I20260926 09:27:33.167478 2812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59335
I20260926 09:27:33.168718 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 09:27:33.179348 2851 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:33.178495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38710 (local address 127.0.98.62:46153)
0926 09:27:33.178568 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:33.178571 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:33.178663 (+ 92us) server_negotiation.cc:407] Connection header received
0926 09:27:33.178713 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:33.178713 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:33.178723 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:33.178781 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:33.179286 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38710: BlockingRecv error: recv got EOF from 127.0.0.1:38710 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 09:27:33.180140 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 2874
I20260926 09:27:33.185245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 3008
I20260926 09:27:33.189457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 3142
I20260926 09:27:33.193404 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 2780
2026-09-26T09:27:33Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1810 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:27:33 dist-test-slave-dfzm krb5kdc[3283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:27:33 dist-test-slave-dfzm krb5kdc[3283](info): set up 2 sockets
Sep 26 09:27:33 dist-test-slave-dfzm krb5kdc[3283](info): commencing operation
krb5kdc: starting...
W20260926 09:27:34.233433 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:27:34 dist-test-slave-dfzm krb5kdc[3283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:27:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:27:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:27:34.346114 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38589
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:34.429664 3296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:34.429940 3296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:34.429982 3296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:34.431192 3296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:27:34.431269 3296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:34.431293 3296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:34.431313 3296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:27:34.431370 3296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:27:34.432947 3296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:34.433244 3296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:34.433506 3296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:34.438406 3301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:34.438393 3302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:34.438393 3304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:34.438706 3296 server_base.cc:1034] running on GCE node
I20260926 09:27:34.438951 3296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:34.439242 3296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:34.440404 3296 hybrid_clock.cc:648] HybridClock initialized: now 1790414854440400 us; error 38 us; skew 500 ppm
Sep 26 09:27:34 dist-test-slave-dfzm krb5kdc[3283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:34.442894 3296 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:27:34.443360 3296 webserver.cc:460] Webserver started at http://127.0.98.62:46263/ using document root <none> and password file <none>
I20260926 09:27:34.443557 3296 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:34.443598 3296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:34.443686 3296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:34.444743 3296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "69c0b99e635d4171b7fcfc72901e5830"
format_stamp: "Formatted at 2026-09-26 09:27:34 on dist-test-slave-dfzm"
server_key: "122beb4e93b3abeddf67fe651affe2cc"
server_key_iv: "6418fa38a5370ce3ff2fe672a9fd6347"
server_key_version: "encryptionkey@0"
I20260926 09:27:34.445040 3296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "69c0b99e635d4171b7fcfc72901e5830"
format_stamp: "Formatted at 2026-09-26 09:27:34 on dist-test-slave-dfzm"
server_key: "122beb4e93b3abeddf67fe651affe2cc"
server_key_iv: "6418fa38a5370ce3ff2fe672a9fd6347"
server_key_version: "encryptionkey@0"
I20260926 09:27:34.446372 3296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:27:34.447077 3311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:34.447275 3296 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:27:34.447364 3296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "69c0b99e635d4171b7fcfc72901e5830"
format_stamp: "Formatted at 2026-09-26 09:27:34 on dist-test-slave-dfzm"
server_key: "122beb4e93b3abeddf67fe651affe2cc"
server_key_iv: "6418fa38a5370ce3ff2fe672a9fd6347"
server_key_version: "encryptionkey@0"
I20260926 09:27:34.447438 3296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:34.455060 3296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:34.456844 3296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:34.456995 3296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:34.460969 3296 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38589
I20260926 09:27:34.460999 3363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38589 every 8 connection(s)
I20260926 09:27:34.461354 3296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:27:34.462251 3364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:34.464461 3364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c0b99e635d4171b7fcfc72901e5830: Bootstrap starting.
I20260926 09:27:34.465044 3364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69c0b99e635d4171b7fcfc72901e5830: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:34.465425 3364 log.cc:824] T 00000000000000000000000000000000 P 69c0b99e635d4171b7fcfc72901e5830: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:34.466096 3364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c0b99e635d4171b7fcfc72901e5830: No bootstrap required, opened a new log
I20260926 09:27:34.467578 3364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69c0b99e635d4171b7fcfc72901e5830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c0b99e635d4171b7fcfc72901e5830" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38589 } }
I20260926 09:27:34.467732 3364 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69c0b99e635d4171b7fcfc72901e5830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:34.467756 3364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69c0b99e635d4171b7fcfc72901e5830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69c0b99e635d4171b7fcfc72901e5830, State: Initialized, Role: FOLLOWER
I20260926 09:27:34.467891 3364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69c0b99e635d4171b7fcfc72901e5830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c0b99e635d4171b7fcfc72901e5830" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38589 } }
I20260926 09:27:34.467967 3364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69c0b99e635d4171b7fcfc72901e5830 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:27:34.467998 3364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69c0b99e635d4171b7fcfc72901e5830 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:27:34.468025 3364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69c0b99e635d4171b7fcfc72901e5830 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:34.468750 3364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69c0b99e635d4171b7fcfc72901e5830 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c0b99e635d4171b7fcfc72901e5830" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38589 } }
I20260926 09:27:34.468832 3364 leader_election.cc:302] T 00000000000000000000000000000000 P 69c0b99e635d4171b7fcfc72901e5830 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69c0b99e635d4171b7fcfc72901e5830; no voters:
I20260926 09:27:34.468945 3364 leader_election.cc:288] T 00000000000000000000000000000000 P 69c0b99e635d4171b7fcfc72901e5830 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:27:34.469009 3367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69c0b99e635d4171b7fcfc72901e5830 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:34.469201 3364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69c0b99e635d4171b7fcfc72901e5830 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:27:34.469234 3367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69c0b99e635d4171b7fcfc72901e5830 [term 1 LEADER]: Becoming Leader. State: Replica: 69c0b99e635d4171b7fcfc72901e5830, State: Running, Role: LEADER
I20260926 09:27:34.469329 3367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69c0b99e635d4171b7fcfc72901e5830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c0b99e635d4171b7fcfc72901e5830" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38589 } }
I20260926 09:27:34.469782 3368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c0b99e635d4171b7fcfc72901e5830 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69c0b99e635d4171b7fcfc72901e5830" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c0b99e635d4171b7fcfc72901e5830" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38589 } } }
I20260926 09:27:34.469882 3368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c0b99e635d4171b7fcfc72901e5830 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:34.470054 3369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c0b99e635d4171b7fcfc72901e5830 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69c0b99e635d4171b7fcfc72901e5830. Latest consensus state: current_term: 1 leader_uuid: "69c0b99e635d4171b7fcfc72901e5830" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c0b99e635d4171b7fcfc72901e5830" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38589 } } }
I20260926 09:27:34.470108 3369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c0b99e635d4171b7fcfc72901e5830 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:34.470494 3376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:27:34.470824 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 3296
I20260926 09:27:34.470935 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:27:34.471108 392 external_mini_cluster.cc:1428] Setting key 3801c164b99981c7f54dd44f30d5c8e6
I20260926 09:27:34.471021 3376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:27:34.472801 3376 catalog_manager.cc:1347] Generated new cluster ID: 88fd7394733440e9b51cdb993829336d
I20260926 09:27:34.472853 3376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:27:34 dist-test-slave-dfzm krb5kdc[3283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:34.478951 3376 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:27:34.479652 3376 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:27:34.481880 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:34.471581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46744 (local address 127.0.98.62:38589)
0926 09:27:34.472215 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:34.472224 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:34.472253 (+ 29us) server_negotiation.cc:407] Connection header received
0926 09:27:34.472365 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:34.472371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:34.472423 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:34.472561 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:34.473284 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:34.474308 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:34.475086 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:34.475545 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:34.475667 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:34.477996 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:34.478000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:34.478001 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:34.478014 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:34.479863 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:34.480223 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:34.480226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:34.480226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:34.480319 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:34.480683 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:34.480684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:34.480685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:34.480953 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:34.481333 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:34.481408 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 09:27:34.481712 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":79,"threads_started":1}
I20260926 09:27:34.482785 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:27:34.484126 3376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69c0b99e635d4171b7fcfc72901e5830: Generated new TSK 0
I20260926 09:27:34.484292 3376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:27:34.496845 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:34.493011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46746 (local address 127.0.98.62:38589)
0926 09:27:34.493078 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:34.493081 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:34.493228 (+ 147us) server_negotiation.cc:407] Connection header received
0926 09:27:34.493300 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:34.493301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:34.493311 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:34.493362 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:34.493841 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:34.494345 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:34.495444 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:34.495659 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:34.495713 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:34.496385 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:34.496386 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:34.496386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:27:34.496573 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:34.496589 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:27:34.496595 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:34.496640 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:34.496726 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 09:27:34.496750 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:27:34.499270 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 3296
2026-09-26T09:27:34Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1303 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:27:34 dist-test-slave-dfzm krb5kdc[3397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:27:34 dist-test-slave-dfzm krb5kdc[3397](info): set up 2 sockets
Sep 26 09:27:34 dist-test-slave-dfzm krb5kdc[3397](info): commencing operation
krb5kdc: starting...
W20260926 09:27:35.545552 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:27:35 dist-test-slave-dfzm krb5kdc[3397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:27:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:27:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:27:35.662746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42379
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:35.742743 3410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:35.742970 3410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:35.742995 3410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:35.744122 3410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:27:35.744164 3410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:35.744179 3410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:35.744191 3410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:27:35.744203 3410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:27:35.745729 3410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:35.745963 3410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:35.746146 3410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:35.751080 3416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:35.751086 3418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:35.751080 3415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:35.751396 3410 server_base.cc:1034] running on GCE node
I20260926 09:27:35.751655 3410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:35.751946 3410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:35.753087 3410 hybrid_clock.cc:648] HybridClock initialized: now 1790414855753073 us; error 29 us; skew 500 ppm
Sep 26 09:27:35 dist-test-slave-dfzm krb5kdc[3397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:35.755751 3410 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:27:35.756234 3410 webserver.cc:460] Webserver started at http://127.0.98.62:39091/ using document root <none> and password file <none>
I20260926 09:27:35.756477 3410 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:35.756521 3410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:35.756681 3410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:35.757825 3410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "4f29be5a428e4ab39dd05104b5ec954f"
format_stamp: "Formatted at 2026-09-26 09:27:35 on dist-test-slave-dfzm"
server_key: "25fe725cc170d6ea49dceb31067ebdbe"
server_key_iv: "fc1b2c2e4f49ffadb4fa6f5dbf4e7674"
server_key_version: "encryptionkey@0"
I20260926 09:27:35.758193 3410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "4f29be5a428e4ab39dd05104b5ec954f"
format_stamp: "Formatted at 2026-09-26 09:27:35 on dist-test-slave-dfzm"
server_key: "25fe725cc170d6ea49dceb31067ebdbe"
server_key_iv: "fc1b2c2e4f49ffadb4fa6f5dbf4e7674"
server_key_version: "encryptionkey@0"
I20260926 09:27:35.759469 3410 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:35.760203 3425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:35.760412 3410 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:35.760488 3410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "4f29be5a428e4ab39dd05104b5ec954f"
format_stamp: "Formatted at 2026-09-26 09:27:35 on dist-test-slave-dfzm"
server_key: "25fe725cc170d6ea49dceb31067ebdbe"
server_key_iv: "fc1b2c2e4f49ffadb4fa6f5dbf4e7674"
server_key_version: "encryptionkey@0"
I20260926 09:27:35.760579 3410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:35.779497 3410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:35.781446 3410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:35.781641 3410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:35.785909 3410 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42379
I20260926 09:27:35.785986 3477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42379 every 8 connection(s)
I20260926 09:27:35.786350 3410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:27:35.787334 3478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:35.787691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 3410
I20260926 09:27:35.787774 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:27:35.788031 392 external_mini_cluster.cc:1428] Setting key 0fd45876eb5afcc063f6c11b2c549794
I20260926 09:27:35.789978 3478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f29be5a428e4ab39dd05104b5ec954f: Bootstrap starting.
I20260926 09:27:35.790548 3478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f29be5a428e4ab39dd05104b5ec954f: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:35.790838 3478 log.cc:824] T 00000000000000000000000000000000 P 4f29be5a428e4ab39dd05104b5ec954f: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:35.791510 3478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f29be5a428e4ab39dd05104b5ec954f: No bootstrap required, opened a new log
I20260926 09:27:35.793412 3478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f29be5a428e4ab39dd05104b5ec954f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f29be5a428e4ab39dd05104b5ec954f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42379 } }
I20260926 09:27:35.793572 3478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f29be5a428e4ab39dd05104b5ec954f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:35.793646 3478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f29be5a428e4ab39dd05104b5ec954f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f29be5a428e4ab39dd05104b5ec954f, State: Initialized, Role: FOLLOWER
I20260926 09:27:35.793802 3478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f29be5a428e4ab39dd05104b5ec954f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f29be5a428e4ab39dd05104b5ec954f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42379 } }
I20260926 09:27:35.793915 3478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f29be5a428e4ab39dd05104b5ec954f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:27:35.793960 3478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f29be5a428e4ab39dd05104b5ec954f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:27:35.794008 3478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f29be5a428e4ab39dd05104b5ec954f [term 0 FOLLOWER]: Advancing to term 1
Sep 26 09:27:35 dist-test-slave-dfzm krb5kdc[3397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:35.794570 3478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f29be5a428e4ab39dd05104b5ec954f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f29be5a428e4ab39dd05104b5ec954f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42379 } }
I20260926 09:27:35.794684 3478 leader_election.cc:302] T 00000000000000000000000000000000 P 4f29be5a428e4ab39dd05104b5ec954f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f29be5a428e4ab39dd05104b5ec954f; no voters:
I20260926 09:27:35.794874 3478 leader_election.cc:288] T 00000000000000000000000000000000 P 4f29be5a428e4ab39dd05104b5ec954f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:27:35.794937 3483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f29be5a428e4ab39dd05104b5ec954f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:35.795128 3483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f29be5a428e4ab39dd05104b5ec954f [term 1 LEADER]: Becoming Leader. State: Replica: 4f29be5a428e4ab39dd05104b5ec954f, State: Running, Role: LEADER
I20260926 09:27:35.795137 3478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f29be5a428e4ab39dd05104b5ec954f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:27:35.795233 3483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f29be5a428e4ab39dd05104b5ec954f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f29be5a428e4ab39dd05104b5ec954f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42379 } }
I20260926 09:27:35.795565 3484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f29be5a428e4ab39dd05104b5ec954f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f29be5a428e4ab39dd05104b5ec954f. Latest consensus state: current_term: 1 leader_uuid: "4f29be5a428e4ab39dd05104b5ec954f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f29be5a428e4ab39dd05104b5ec954f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42379 } } }
I20260926 09:27:35.795644 3484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f29be5a428e4ab39dd05104b5ec954f [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:35.795671 3485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f29be5a428e4ab39dd05104b5ec954f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f29be5a428e4ab39dd05104b5ec954f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f29be5a428e4ab39dd05104b5ec954f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42379 } } }
I20260926 09:27:35.795814 3485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f29be5a428e4ab39dd05104b5ec954f [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:35.795864 3488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:27:35.796386 3488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:27:35.798013 3488 catalog_manager.cc:1347] Generated new cluster ID: 2c5231df67f049ef8b363a90c0178d57
I20260926 09:27:35.798066 3488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:27:35.799721 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:35.788607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52774 (local address 127.0.98.62:42379)
0926 09:27:35.788789 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:35.788797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:35.788822 (+ 25us) server_negotiation.cc:407] Connection header received
0926 09:27:35.789197 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:35.789203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:35.789247 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:35.789412 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:35.790055 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:35.791077 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:35.791917 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:35.792347 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:35.792475 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:35.795033 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:35.795037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:35.795037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:35.795049 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:35.798194 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:35.798566 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:35.798570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:35.798570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:35.798667 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:35.798892 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:35.798894 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:35.798894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:35.799134 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:35.799278 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:35.799413 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 09:27:35.799549 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":77,"threads_started":1}
I20260926 09:27:35.813867 3488 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:27:35.814450 3488 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:27:35.819422 3488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f29be5a428e4ab39dd05104b5ec954f: Generated new TSK 0
I20260926 09:27:35.819645 3488 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:27:35.852912 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42379
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:35.969517 3504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:35.969822 3504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:35.969857 3504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:35.971632 3504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:35.971689 3504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:35.971724 3504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:27:35.973650 3504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:35.973906 3504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:35.974159 3504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:35.979461 3512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:35.979563 3504 server_base.cc:1034] running on GCE node
W20260926 09:27:35.979636 3509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:35.979461 3510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:35.979900 3504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:35.980115 3504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:35.981252 3504 hybrid_clock.cc:648] HybridClock initialized: now 1790414855981239 us; error 28 us; skew 500 ppm
Sep 26 09:27:35 dist-test-slave-dfzm krb5kdc[3397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:35.983759 3504 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:27:35.984213 3504 webserver.cc:460] Webserver started at http://127.0.98.1:41633/ using document root <none> and password file <none>
I20260926 09:27:35.984447 3504 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:35.984493 3504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:35.984633 3504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:35.985756 3504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "cd72cabc2fe34f18811b43c3ab9a8bfa"
format_stamp: "Formatted at 2026-09-26 09:27:35 on dist-test-slave-dfzm"
server_key: "1f85742eb7c44619bdf90629e0b8988b"
server_key_iv: "57f59be58b129f37980b854584123716"
server_key_version: "encryptionkey@0"
I20260926 09:27:35.986109 3504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cd72cabc2fe34f18811b43c3ab9a8bfa"
format_stamp: "Formatted at 2026-09-26 09:27:35 on dist-test-slave-dfzm"
server_key: "1f85742eb7c44619bdf90629e0b8988b"
server_key_iv: "57f59be58b129f37980b854584123716"
server_key_version: "encryptionkey@0"
I20260926 09:27:35.987406 3504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:27:35.988118 3519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:35.988291 3504 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:27:35.988406 3504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "cd72cabc2fe34f18811b43c3ab9a8bfa"
format_stamp: "Formatted at 2026-09-26 09:27:35 on dist-test-slave-dfzm"
server_key: "1f85742eb7c44619bdf90629e0b8988b"
server_key_iv: "57f59be58b129f37980b854584123716"
server_key_version: "encryptionkey@0"
I20260926 09:27:35.988492 3504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:36.006263 3504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:36.008136 3504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:36.008311 3504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:36.008575 3504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:36.008894 3504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:36.008948 3504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:36.008993 3504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:36.009018 3504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:36.014840 3504 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42167
I20260926 09:27:36.014889 3632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42167 every 8 connection(s)
I20260926 09:27:36.015208 3504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:27:36.018599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 3504
I20260926 09:27:36.018733 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:27:36.018945 392 external_mini_cluster.cc:1428] Setting key 35af5e049dee6c3397d32c03ca92b2a1
Sep 26 09:27:36 dist-test-slave-dfzm krb5kdc[3397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:36.026082 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:36.015940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38087 (local address 127.0.98.62:42379)
0926 09:27:36.016012 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:36.016014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:36.016215 (+ 201us) server_negotiation.cc:407] Connection header received
0926 09:27:36.016756 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:36.016757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:36.016767 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:36.016824 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:36.017911 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:36.018471 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:36.019863 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:36.020214 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:36.020305 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:36.022807 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:36.022809 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:36.022809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:36.022817 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:36.024627 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:36.025017 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:36.025018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:36.025019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:36.025084 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:36.025318 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:36.025319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:36.025320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:36.025479 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:36.025554 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:36.025864 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 09:27:36.025975 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 09:27:36.026096 3635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:36.015873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42379 (local address 127.0.98.1:38087)
0926 09:27:36.016075 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:36.016117 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:36.016653 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:36.016897 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:36.016900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:36.017481 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:36.017790 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:36.017797 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:36.018573 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:36.018573 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:36.019770 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:36.019777 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:36.020307 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:36.020308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:36.020424 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:36.021060 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:36.021072 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:36.022716 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:36.024710 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:36.024716 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:36.024717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:36.024938 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:36.025156 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:36.025157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:36.025157 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:36.025252 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:36.025549 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:36.025552 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:36.025724 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 09:27:36.025810 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 09:27:36.025932 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":113,"threads_started":1}
I20260926 09:27:36.026659 3633 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42379
I20260926 09:27:36.026788 3633 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:36.026973 3633 heartbeater.cc:511] Master 127.0.98.62:42379 requested a full tablet report, sending...
I20260926 09:27:36.027540 3442 ts_manager.cc:194] Registered new tserver with Master: cd72cabc2fe34f18811b43c3ab9a8bfa (127.0.98.1:42167)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 09:27:36.028481 3442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:27:36.044601 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42379
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:36.125049 3638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:36.125299 3638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:36.125340 3638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:36.126510 3638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:36.126583 3638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:36.126628 3638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:27:36.128057 3638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:36.128329 3638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:36.128558 3638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:36.133545 3643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:36.133543 3646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:36.133682 3638 server_base.cc:1034] running on GCE node
W20260926 09:27:36.133688 3644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:36.134079 3638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:36.134351 3638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:36.135504 3638 hybrid_clock.cc:648] HybridClock initialized: now 1790414856135485 us; error 30 us; skew 500 ppm
Sep 26 09:27:36 dist-test-slave-dfzm krb5kdc[3397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:36.138000 3638 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:27:36.138432 3638 webserver.cc:460] Webserver started at http://127.0.98.2:36587/ using document root <none> and password file <none>
I20260926 09:27:36.138666 3638 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:36.138734 3638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:36.138902 3638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:36.139943 3638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "bd5dc249b0964f9e824d1e5233d526e2"
format_stamp: "Formatted at 2026-09-26 09:27:36 on dist-test-slave-dfzm"
server_key: "00e2c65bde20fa6854877998d3256661"
server_key_iv: "a4efeb9ae5d66607e6789e5b34ae2827"
server_key_version: "encryptionkey@0"
I20260926 09:27:36.140300 3638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bd5dc249b0964f9e824d1e5233d526e2"
format_stamp: "Formatted at 2026-09-26 09:27:36 on dist-test-slave-dfzm"
server_key: "00e2c65bde20fa6854877998d3256661"
server_key_iv: "a4efeb9ae5d66607e6789e5b34ae2827"
server_key_version: "encryptionkey@0"
I20260926 09:27:36.141556 3638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:27:36.142282 3653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:36.142444 3638 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:36.142539 3638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "bd5dc249b0964f9e824d1e5233d526e2"
format_stamp: "Formatted at 2026-09-26 09:27:36 on dist-test-slave-dfzm"
server_key: "00e2c65bde20fa6854877998d3256661"
server_key_iv: "a4efeb9ae5d66607e6789e5b34ae2827"
server_key_version: "encryptionkey@0"
I20260926 09:27:36.142620 3638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:36.159415 3638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:36.161257 3638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:36.161415 3638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:36.161707 3638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:36.162021 3638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:36.162075 3638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:36.162117 3638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:36.162143 3638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:36.168417 3638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35125
I20260926 09:27:36.168484 3766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35125 every 8 connection(s)
I20260926 09:27:36.168849 3638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:27:36.169448 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 3638
I20260926 09:27:36.169567 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:27:36.169783 392 external_mini_cluster.cc:1428] Setting key 2ac8ec71f40ad0427ead53b2f90f4c4b
Sep 26 09:27:36 dist-test-slave-dfzm krb5kdc[3397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 09:27:36.178953 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:36.169612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59779 (local address 127.0.98.62:42379)
0926 09:27:36.169660 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:36.169662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:36.169916 (+ 254us) server_negotiation.cc:407] Connection header received
0926 09:27:36.170423 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:36.170425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:36.170436 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:36.170490 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:36.171431 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:36.171931 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:36.173299 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:36.173492 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:36.173553 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:36.176084 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:36.176086 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:36.176086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:36.176094 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:36.177647 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:36.178062 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:36.178064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:36.178064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:36.178124 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:36.178373 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:36.178374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:36.178374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:36.178523 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:36.178590 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:36.178822 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 09:27:36.178884 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:27:36.179124 3769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:36.169619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42379 (local address 127.0.98.2:59779)
0926 09:27:36.169780 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:36.169822 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:36.170313 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:36.170569 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:36.170573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:36.171143 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:36.171327 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:36.171332 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:36.172030 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:36.172031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:36.173210 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:36.173213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:36.173606 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:36.173607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:36.173762 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:36.174303 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:36.174313 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:36.175989 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:36.177742 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:36.177749 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:36.177751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:36.177971 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:36.178191 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:36.178192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:36.178192 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:36.178304 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:36.178593 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:36.178598 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:36.178716 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 09:27:36.178809 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 09:27:36.178928 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260926 09:27:36.179680 3767 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42379
I20260926 09:27:36.179772 3767 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:36.179911 3767 heartbeater.cc:511] Master 127.0.98.62:42379 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 09:27:36.180327 3442 ts_manager.cc:194] Registered new tserver with Master: bd5dc249b0964f9e824d1e5233d526e2 (127.0.98.2:35125)
I20260926 09:27:36.180699 3442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:27:36.195787 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42379
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:36.275277 3772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:36.275549 3772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:36.275588 3772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:36.276767 3772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:36.276839 3772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:36.276893 3772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:27:36.278373 3772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:36.278643 3772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:36.278882 3772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:27:36.284042 3772 server_base.cc:1034] running on GCE node
W20260926 09:27:36.284003 3778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:36.284053 3780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:36.284039 3777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:36.284403 3772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:36.284607 3772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:36.285727 3772 hybrid_clock.cc:648] HybridClock initialized: now 1790414856285715 us; error 30 us; skew 500 ppm
Sep 26 09:27:36 dist-test-slave-dfzm krb5kdc[3397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:36.288275 3772 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:27:36.288751 3772 webserver.cc:460] Webserver started at http://127.0.98.3:46315/ using document root <none> and password file <none>
I20260926 09:27:36.288980 3772 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:36.289047 3772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:36.289198 3772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:36.290341 3772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "576ddc32f63b4b7e8830a33858a2c2cf"
format_stamp: "Formatted at 2026-09-26 09:27:36 on dist-test-slave-dfzm"
server_key: "dace8736201217c27ecf55ed8f8cc67a"
server_key_iv: "f1ffd217d9489b250dc0afff2f843202"
server_key_version: "encryptionkey@0"
I20260926 09:27:36.290709 3772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "576ddc32f63b4b7e8830a33858a2c2cf"
format_stamp: "Formatted at 2026-09-26 09:27:36 on dist-test-slave-dfzm"
server_key: "dace8736201217c27ecf55ed8f8cc67a"
server_key_iv: "f1ffd217d9489b250dc0afff2f843202"
server_key_version: "encryptionkey@0"
I20260926 09:27:36.292187 3772 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:27:36.293057 3787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:36.293242 3772 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:27:36.293351 3772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "576ddc32f63b4b7e8830a33858a2c2cf"
format_stamp: "Formatted at 2026-09-26 09:27:36 on dist-test-slave-dfzm"
server_key: "dace8736201217c27ecf55ed8f8cc67a"
server_key_iv: "f1ffd217d9489b250dc0afff2f843202"
server_key_version: "encryptionkey@0"
I20260926 09:27:36.293442 3772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:36.316131 3772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:36.318456 3772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:36.318645 3772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:36.318912 3772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:36.319253 3772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:36.319314 3772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:36.319361 3772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:36.319389 3772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:36.325960 3772 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41317
I20260926 09:27:36.326042 3900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41317 every 8 connection(s)
I20260926 09:27:36.326404 3772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:27:36.330691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 3772
I20260926 09:27:36.330788 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:27:36.330987 392 external_mini_cluster.cc:1428] Setting key f0e4ad1c0a383de854e57fc7a5a6ec50
Sep 26 09:27:36 dist-test-slave-dfzm krb5kdc[3397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:36.336479 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:36.327208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52427 (local address 127.0.98.62:42379)
0926 09:27:36.327287 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:36.327291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:36.327487 (+ 196us) server_negotiation.cc:407] Connection header received
0926 09:27:36.328077 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:36.328078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:36.328091 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:36.328142 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:36.329105 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:36.329657 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:36.330955 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:36.331213 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:36.331281 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:36.333578 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:36.333579 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:36.333580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:36.333629 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:36.335251 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:36.335641 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:36.335642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:36.335642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:36.335695 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:36.335912 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:36.335913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:36.335913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:36.336017 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:36.336069 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:36.336344 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 09:27:36.336414 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 09:27:36.336656 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:36.327184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42379 (local address 127.0.98.3:52427)
0926 09:27:36.327363 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:36.327400 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:36.327955 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:36.328216 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:36.328220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:36.328788 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:36.328993 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:36.328998 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:36.329753 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:36.329754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:36.330811 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:36.330814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:36.331290 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:36.331291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:36.331453 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:36.332042 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:36.332053 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:36.333484 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:36.335332 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:36.335338 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:36.335340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:36.335573 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:36.335755 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:36.335756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:36.335756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:36.335853 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:36.336073 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:36.336077 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:36.336198 (+ 121us) client_negotiation.cc:769] Sending connection context
0926 09:27:36.336283 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 09:27:36.336422 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
I20260926 09:27:36.337226 3901 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42379
I20260926 09:27:36.337330 3901 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:36.337481 3901 heartbeater.cc:511] Master 127.0.98.62:42379 requested a full tablet report, sending...
I20260926 09:27:36.337858 3442 ts_manager.cc:194] Registered new tserver with Master: 576ddc32f63b4b7e8830a33858a2c2cf (127.0.98.3:41317)
I20260926 09:27:36.338240 3442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52427
I20260926 09:27:36.343132 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 09:27:36 dist-test-slave-dfzm krb5kdc[3397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414856, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 09:27:36 dist-test-slave-dfzm krb5kdc[3397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414856, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:36.365525 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:36.358610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52782 (local address 127.0.98.62:42379)
0926 09:27:36.358825 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:36.358827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:36.358837 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:27:36.358863 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:36.358864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:36.358873 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:36.358950 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:36.359509 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:36.360026 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:36.360781 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:36.360960 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:36.361014 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:36.363339 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:36.363340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:36.363340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:36.363347 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:36.364577 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:36.364861 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:36.364862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:36.364862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:36.364927 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:36.365071 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:36.365072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:36.365072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:36.365198 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:36.365266 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:36.365358 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 09:27:36.365417 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
W20260926 09:27:36.365804 3442 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52782
I20260926 09:27:36.366351 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 3504
I20260926 09:27:36.371282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 3638
I20260926 09:27:36.375836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 3772
I20260926 09:27:36.380403 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 3410
2026-09-26T09:27:36Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1884 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:27:36 dist-test-slave-dfzm krb5kdc[3913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:27:36 dist-test-slave-dfzm krb5kdc[3913](info): set up 2 sockets
Sep 26 09:27:36 dist-test-slave-dfzm krb5kdc[3913](info): commencing operation
krb5kdc: starting...
W20260926 09:27:37.453491 392 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:27:37 dist-test-slave-dfzm krb5kdc[3913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:27:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:27:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:27:37.570933 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34889
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34889
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:37.654634 3926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:37.654836 3926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:37.654858 3926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:37.656090 3926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:27:37.656132 3926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:37.656147 3926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:37.656159 3926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:27:37.656172 3926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:27:37.657645 3926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:37.657887 3926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:37.658121 3926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:37.662889 3932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:37.662882 3934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:37.662911 3931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:37.663077 3926 server_base.cc:1034] running on GCE node
I20260926 09:27:37.663364 3926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:37.663664 3926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:37.664839 3926 hybrid_clock.cc:648] HybridClock initialized: now 1790414857664817 us; error 35 us; skew 500 ppm
Sep 26 09:27:37 dist-test-slave-dfzm krb5kdc[3913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:37.671576 3926 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:27:37.672065 3926 webserver.cc:460] Webserver started at http://127.0.98.62:40063/ using document root <none> and password file <none>
I20260926 09:27:37.672294 3926 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:37.672361 3926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:37.672516 3926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:37.674062 3926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "98a5981d63d94039b438d154b8e6fc81"
format_stamp: "Formatted at 2026-09-26 09:27:37 on dist-test-slave-dfzm"
server_key: "7f18ee31e59fb540711f3fc374161a81"
server_key_iv: "028cba731b7403e77de35e3bf0a3ec7a"
server_key_version: "encryptionkey@0"
I20260926 09:27:37.674423 3926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "98a5981d63d94039b438d154b8e6fc81"
format_stamp: "Formatted at 2026-09-26 09:27:37 on dist-test-slave-dfzm"
server_key: "7f18ee31e59fb540711f3fc374161a81"
server_key_iv: "028cba731b7403e77de35e3bf0a3ec7a"
server_key_version: "encryptionkey@0"
I20260926 09:27:37.675714 3926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:27:37.676450 3941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:37.676620 3926 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:37.676715 3926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "98a5981d63d94039b438d154b8e6fc81"
format_stamp: "Formatted at 2026-09-26 09:27:37 on dist-test-slave-dfzm"
server_key: "7f18ee31e59fb540711f3fc374161a81"
server_key_iv: "028cba731b7403e77de35e3bf0a3ec7a"
server_key_version: "encryptionkey@0"
I20260926 09:27:37.676797 3926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:37.685220 3926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:37.687139 3926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:37.687289 3926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:37.691035 3926 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34889
I20260926 09:27:37.691069 3993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34889 every 8 connection(s)
I20260926 09:27:37.691449 3926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:27:37.692305 3994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:37.694555 3994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a5981d63d94039b438d154b8e6fc81: Bootstrap starting.
I20260926 09:27:37.695065 3994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98a5981d63d94039b438d154b8e6fc81: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:37.695297 3994 log.cc:824] T 00000000000000000000000000000000 P 98a5981d63d94039b438d154b8e6fc81: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:37.695871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 3926
I20260926 09:27:37.695899 3994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a5981d63d94039b438d154b8e6fc81: No bootstrap required, opened a new log
I20260926 09:27:37.695992 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:27:37.696190 392 external_mini_cluster.cc:1428] Setting key 5532c41bcfb59f6a5b3515e95e3c30ab
I20260926 09:27:37.698484 3994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98a5981d63d94039b438d154b8e6fc81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a5981d63d94039b438d154b8e6fc81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34889 } }
I20260926 09:27:37.698657 3994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98a5981d63d94039b438d154b8e6fc81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:37.698694 3994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98a5981d63d94039b438d154b8e6fc81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a5981d63d94039b438d154b8e6fc81, State: Initialized, Role: FOLLOWER
I20260926 09:27:37.698819 3994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98a5981d63d94039b438d154b8e6fc81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a5981d63d94039b438d154b8e6fc81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34889 } }
I20260926 09:27:37.698920 3994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98a5981d63d94039b438d154b8e6fc81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:27:37.698963 3994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98a5981d63d94039b438d154b8e6fc81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:27:37.699011 3994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98a5981d63d94039b438d154b8e6fc81 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:37.699596 3994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98a5981d63d94039b438d154b8e6fc81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a5981d63d94039b438d154b8e6fc81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34889 } }
I20260926 09:27:37.699702 3994 leader_election.cc:302] T 00000000000000000000000000000000 P 98a5981d63d94039b438d154b8e6fc81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98a5981d63d94039b438d154b8e6fc81; no voters:
I20260926 09:27:37.699854 3994 leader_election.cc:288] T 00000000000000000000000000000000 P 98a5981d63d94039b438d154b8e6fc81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:27:37.699929 3999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98a5981d63d94039b438d154b8e6fc81 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:37.700099 3999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98a5981d63d94039b438d154b8e6fc81 [term 1 LEADER]: Becoming Leader. State: Replica: 98a5981d63d94039b438d154b8e6fc81, State: Running, Role: LEADER
I20260926 09:27:37.700107 3994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98a5981d63d94039b438d154b8e6fc81 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:27:37.700201 3999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98a5981d63d94039b438d154b8e6fc81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a5981d63d94039b438d154b8e6fc81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34889 } }
I20260926 09:27:37.700598 4000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98a5981d63d94039b438d154b8e6fc81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98a5981d63d94039b438d154b8e6fc81. Latest consensus state: current_term: 1 leader_uuid: "98a5981d63d94039b438d154b8e6fc81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a5981d63d94039b438d154b8e6fc81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34889 } } }
I20260926 09:27:37.700672 4000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98a5981d63d94039b438d154b8e6fc81 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:37.700882 4005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:27:37.701359 4005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:27:37.701761 4001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98a5981d63d94039b438d154b8e6fc81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98a5981d63d94039b438d154b8e6fc81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a5981d63d94039b438d154b8e6fc81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34889 } } }
I20260926 09:27:37.702157 4001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98a5981d63d94039b438d154b8e6fc81 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:37.703655 4005 catalog_manager.cc:1347] Generated new cluster ID: f3427203b10c488b9d94de1c588cc9f9
I20260926 09:27:37.703709 4005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:27:37 dist-test-slave-dfzm krb5kdc[3913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:37.709117 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:37.696685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48192 (local address 127.0.98.62:34889)
0926 09:27:37.696868 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:37.696879 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:37.696912 (+ 33us) server_negotiation.cc:407] Connection header received
0926 09:27:37.698439 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:37.698448 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:37.698511 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:37.698672 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:37.699439 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:37.700841 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:37.702002 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:37.702633 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:37.702768 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:37.705473 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:37.705477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:37.705478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:37.705489 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:37.707481 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:37.707877 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:37.707881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:37.707881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:37.707985 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:37.708211 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:37.708212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:37.708212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:37.708516 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:37.708675 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:37.708775 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 09:27:37.708910 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260926 09:27:37.709435 4005 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:27:37.710006 4005 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:27:37.718122 4005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98a5981d63d94039b438d154b8e6fc81: Generated new TSK 0
I20260926 09:27:37.718290 4005 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:27:37.745846 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34889
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:37.826630 4020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:37.826826 4020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:37.826850 4020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:37.827915 4020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:37.827950 4020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:37.827975 4020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:27:37.829357 4020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:37.829536 4020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:37.829774 4020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:37.834772 4026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:37.834795 4025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:37.834787 4028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:37.835170 4020 server_base.cc:1034] running on GCE node
I20260926 09:27:37.835402 4020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:37.835675 4020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:37.836830 4020 hybrid_clock.cc:648] HybridClock initialized: now 1790414857836824 us; error 51 us; skew 500 ppm
Sep 26 09:27:37 dist-test-slave-dfzm krb5kdc[3913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:37.839461 4020 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:27:37.839910 4020 webserver.cc:460] Webserver started at http://127.0.98.1:36517/ using document root <none> and password file <none>
I20260926 09:27:37.840158 4020 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:37.840229 4020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:37.840376 4020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:37.841406 4020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "32bb3714e44c408381c496abea0ec7cc"
format_stamp: "Formatted at 2026-09-26 09:27:37 on dist-test-slave-dfzm"
server_key: "46ebbc5dec8cc37ed5244120f56d9ba8"
server_key_iv: "ae7baef2aa8f24ab11b01110cf7ba255"
server_key_version: "encryptionkey@0"
I20260926 09:27:37.841782 4020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "32bb3714e44c408381c496abea0ec7cc"
format_stamp: "Formatted at 2026-09-26 09:27:37 on dist-test-slave-dfzm"
server_key: "46ebbc5dec8cc37ed5244120f56d9ba8"
server_key_iv: "ae7baef2aa8f24ab11b01110cf7ba255"
server_key_version: "encryptionkey@0"
I20260926 09:27:37.843067 4020 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:27:37.843782 4035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:37.843943 4020 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:27:37.844048 4020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "32bb3714e44c408381c496abea0ec7cc"
format_stamp: "Formatted at 2026-09-26 09:27:37 on dist-test-slave-dfzm"
server_key: "46ebbc5dec8cc37ed5244120f56d9ba8"
server_key_iv: "ae7baef2aa8f24ab11b01110cf7ba255"
server_key_version: "encryptionkey@0"
I20260926 09:27:37.844123 4020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:37.874701 4020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:37.876643 4020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:37.876818 4020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:37.877063 4020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:37.877382 4020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:37.877439 4020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:37.877480 4020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:37.877509 4020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:37.883505 4020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34941
I20260926 09:27:37.883555 4148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34941 every 8 connection(s)
I20260926 09:27:37.883890 4020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 09:27:37 dist-test-slave-dfzm krb5kdc[3913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:37.890836 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 4020
I20260926 09:27:37.890949 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:27:37.891152 392 external_mini_cluster.cc:1428] Setting key 6cc19677c6a6e954ff0e6b0adf47b182
I20260926 09:27:37.894466 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:37.884684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53091 (local address 127.0.98.62:34889)
0926 09:27:37.884749 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:37.884752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:37.884999 (+ 247us) server_negotiation.cc:407] Connection header received
0926 09:27:37.886025 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:37.886026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:37.886037 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:37.886088 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:37.886692 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:37.887200 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:37.888301 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:37.888730 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:37.888798 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:37.891155 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:37.891157 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:37.891157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:37.891165 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:37.892921 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:37.893414 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:37.893415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:37.893415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:37.893483 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:37.893823 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:37.893823 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:37.893824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:37.893929 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:37.893982 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:37.894329 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 09:27:37.894401 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 09:27:37.894719 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:37.884678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34889 (local address 127.0.98.1:53091)
0926 09:27:37.884867 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:37.884904 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:37.885928 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:37.886155 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:37.886158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:37.886424 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:37.886619 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:37.886623 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:37.887281 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:37.887281 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:37.888206 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:37.888223 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:37.888822 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:37.888823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:37.888961 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:37.889544 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:37.889554 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:37.891035 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:37.893004 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:37.893010 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:37.893012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:37.893319 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:37.893562 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:37.893570 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:37.893570 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:37.893752 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:37.893994 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:37.894000 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:37.894181 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 09:27:37.894302 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 09:27:37.894485 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":92,"threads_started":1}
I20260926 09:27:37.895154 4149 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34889
I20260926 09:27:37.895251 4149 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:37.895448 4149 heartbeater.cc:511] Master 127.0.98.62:34889 requested a full tablet report, sending...
I20260926 09:27:37.895932 3958 ts_manager.cc:194] Registered new tserver with Master: 32bb3714e44c408381c496abea0ec7cc (127.0.98.1:34941)
I20260926 09:27:37.896488 3958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53091
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:27:37.916460 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34889
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:37.996519 4154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:37.996724 4154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:37.996747 4154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:37.997967 4154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:37.998006 4154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:37.998034 4154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:27:37.999627 4154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:37.999854 4154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:38.000077 4154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:38.005081 4159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:38.005167 4154 server_base.cc:1034] running on GCE node
W20260926 09:27:38.005086 4162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:38.005128 4160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:38.005509 4154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:38.005812 4154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:38.006982 4154 hybrid_clock.cc:648] HybridClock initialized: now 1790414858006964 us; error 30 us; skew 500 ppm
Sep 26 09:27:38 dist-test-slave-dfzm krb5kdc[3913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:38.009518 4154 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:27:38.009974 4154 webserver.cc:460] Webserver started at http://127.0.98.2:33465/ using document root <none> and password file <none>
I20260926 09:27:38.010202 4154 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:38.010273 4154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:38.010428 4154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:38.011498 4154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "c8f73f29da684333be4e70e40b8a6e8c"
format_stamp: "Formatted at 2026-09-26 09:27:38 on dist-test-slave-dfzm"
server_key: "83ef5a6ace4dcb2d0f49719ce9087c2a"
server_key_iv: "67d2b57fd50110907886cebd4641ee67"
server_key_version: "encryptionkey@0"
I20260926 09:27:38.011879 4154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c8f73f29da684333be4e70e40b8a6e8c"
format_stamp: "Formatted at 2026-09-26 09:27:38 on dist-test-slave-dfzm"
server_key: "83ef5a6ace4dcb2d0f49719ce9087c2a"
server_key_iv: "67d2b57fd50110907886cebd4641ee67"
server_key_version: "encryptionkey@0"
I20260926 09:27:38.013237 4154 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:27:38.014087 4169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:38.014259 4154 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:38.014364 4154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "c8f73f29da684333be4e70e40b8a6e8c"
format_stamp: "Formatted at 2026-09-26 09:27:38 on dist-test-slave-dfzm"
server_key: "83ef5a6ace4dcb2d0f49719ce9087c2a"
server_key_iv: "67d2b57fd50110907886cebd4641ee67"
server_key_version: "encryptionkey@0"
I20260926 09:27:38.014438 4154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:38.027693 4154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:38.030094 4154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:38.030265 4154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:38.030511 4154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:38.030819 4154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:38.030874 4154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:38.030916 4154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:38.030942 4154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:38.036572 4154 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42587
I20260926 09:27:38.036657 4282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42587 every 8 connection(s)
I20260926 09:27:38.036984 4154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:27:38.041144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 4154
I20260926 09:27:38.041263 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:27:38.041445 392 external_mini_cluster.cc:1428] Setting key a9c57040e467e10725635bb6c3225600
Sep 26 09:27:38 dist-test-slave-dfzm krb5kdc[3913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:38.046988 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:38.037822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50979 (local address 127.0.98.62:34889)
0926 09:27:38.037885 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:38.037887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:38.038065 (+ 178us) server_negotiation.cc:407] Connection header received
0926 09:27:38.039034 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:38.039036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:38.039046 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:38.039120 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:38.039687 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:38.040189 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:38.041163 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:38.041472 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:38.041555 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:38.044019 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:38.044020 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:38.044021 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:38.044029 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:38.045537 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:38.045985 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:38.045986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:38.045987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:38.046043 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:38.046309 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:38.046310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:38.046311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:38.046428 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:38.046498 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:38.046806 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 09:27:38.046895 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 09:27:38.047209 4285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:38.037788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34889 (local address 127.0.98.2:50979)
0926 09:27:38.037939 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:38.037980 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:38.038933 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:38.039165 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:38.039168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:38.039420 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:38.039610 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:38.039614 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:38.040253 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:38.040254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:38.041065 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:38.041066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:38.041648 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:38.041649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:38.041806 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:38.042367 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:38.042375 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:38.043919 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:38.045658 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:38.045665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:38.045667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:38.045893 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:38.046123 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:38.046124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:38.046124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:38.046233 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:38.046501 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:38.046507 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:38.046670 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 09:27:38.046784 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 09:27:38.046943 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260926 09:27:38.047796 4283 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34889
I20260926 09:27:38.047915 4283 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:38.048095 4283 heartbeater.cc:511] Master 127.0.98.62:34889 requested a full tablet report, sending...
I20260926 09:27:38.048525 3958 ts_manager.cc:194] Registered new tserver with Master: c8f73f29da684333be4e70e40b8a6e8c (127.0.98.2:42587)
I20260926 09:27:38.049062 3958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50979
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:27:38.066658 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34889
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:38.147017 4288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:38.147230 4288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:38.147254 4288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:38.148350 4288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:38.148382 4288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:38.148409 4288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:27:38.149788 4288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:38.149971 4288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:38.150158 4288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:38.155045 4296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:38.155047 4293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:38.155165 4294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:38.156277 4288 server_base.cc:1034] running on GCE node
I20260926 09:27:38.156468 4288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:38.156668 4288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:38.157804 4288 hybrid_clock.cc:648] HybridClock initialized: now 1790414858157780 us; error 36 us; skew 500 ppm
Sep 26 09:27:38 dist-test-slave-dfzm krb5kdc[3913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:38.160382 4288 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:27:38.160981 4288 webserver.cc:460] Webserver started at http://127.0.98.3:42613/ using document root <none> and password file <none>
I20260926 09:27:38.161221 4288 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:38.161269 4288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:38.161408 4288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:38.162634 4288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "c707efe3217443fc8ed4e0806e715236"
format_stamp: "Formatted at 2026-09-26 09:27:38 on dist-test-slave-dfzm"
server_key: "d0bb5c973da9591e37761c9e881a14f9"
server_key_iv: "792d28046e033119eb636a8873608a86"
server_key_version: "encryptionkey@0"
I20260926 09:27:38.163013 4288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c707efe3217443fc8ed4e0806e715236"
format_stamp: "Formatted at 2026-09-26 09:27:38 on dist-test-slave-dfzm"
server_key: "d0bb5c973da9591e37761c9e881a14f9"
server_key_iv: "792d28046e033119eb636a8873608a86"
server_key_version: "encryptionkey@0"
I20260926 09:27:38.164404 4288 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:27:38.165136 4303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:38.165318 4288 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:38.165414 4288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "c707efe3217443fc8ed4e0806e715236"
format_stamp: "Formatted at 2026-09-26 09:27:38 on dist-test-slave-dfzm"
server_key: "d0bb5c973da9591e37761c9e881a14f9"
server_key_iv: "792d28046e033119eb636a8873608a86"
server_key_version: "encryptionkey@0"
I20260926 09:27:38.165495 4288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:38.176338 4288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:38.178865 4288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:38.179095 4288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:38.179380 4288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:38.179761 4288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:38.179821 4288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:38.179867 4288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:38.179898 4288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:38.186575 4288 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44359
I20260926 09:27:38.186611 4416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44359 every 8 connection(s)
I20260926 09:27:38.186973 4288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:27:38.191387 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 4288
I20260926 09:27:38.191490 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:27:38.191735 392 external_mini_cluster.cc:1428] Setting key fa9176bd178373341d5c36b4a2303ed3
Sep 26 09:27:38 dist-test-slave-dfzm krb5kdc[3913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:38.197695 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:38.187881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44873 (local address 127.0.98.62:34889)
0926 09:27:38.187955 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:38.187957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:38.188195 (+ 238us) server_negotiation.cc:407] Connection header received
0926 09:27:38.189198 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:38.189199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:38.189208 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:38.189264 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:38.190083 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:38.190568 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:38.191730 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:38.192001 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:38.192123 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:38.194901 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:38.194903 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:38.194903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:38.194911 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:38.196254 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:38.196688 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:38.196690 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:38.196690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:38.196752 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:38.197006 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:38.197007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:38.197007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:38.197124 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:38.197201 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:38.197477 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 09:27:38.197565 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 09:27:38.197882 4419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:38.187863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34889 (local address 127.0.98.3:44873)
0926 09:27:38.188084 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:38.188125 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:38.189101 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:38.189326 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:38.189330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:38.189785 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:38.190000 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:38.190005 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:38.190645 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:38.190646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:38.191638 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:38.191640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:38.192135 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:38.192146 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:38.192320 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:38.193120 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:38.193139 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:38.194793 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:38.196355 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:38.196361 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:38.196363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:38.196588 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:38.196820 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:38.196821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:38.196821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:38.196931 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:38.197192 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:38.197198 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:38.197341 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 09:27:38.197447 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 09:27:38.197641 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":104,"threads_started":1}
I20260926 09:27:38.198359 4417 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34889
I20260926 09:27:38.198457 4417 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:38.198609 4417 heartbeater.cc:511] Master 127.0.98.62:34889 requested a full tablet report, sending...
I20260926 09:27:38.198982 3958 ts_manager.cc:194] Registered new tserver with Master: c707efe3217443fc8ed4e0806e715236 (127.0.98.3:44359)
I20260926 09:27:38.199361 3958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44873
I20260926 09:27:38.203730 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 09:27:38 dist-test-slave-dfzm krb5kdc[3913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 09:27:38 dist-test-slave-dfzm krb5kdc[3913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 09:27:38.226879 4424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:38.219219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58360 (local address 127.0.98.1:34941)
0926 09:27:38.219409 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:38.219413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:38.219424 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:27:38.219451 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:38.219453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:38.219468 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:38.219538 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:38.220077 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:38.220797 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:38.221507 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:38.221894 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:38.221953 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:38.224137 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:38.224145 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:38.224145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:38.224152 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:38.225871 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:38.226182 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:38.226184 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:38.226185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:38.226248 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:38.226409 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:38.226409 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:38.226409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:38.226531 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:38.226637 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:38.226704 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 09:27:38.226763 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":100,"threads_started":1}
Sep 26 09:27:38 dist-test-slave-dfzm krb5kdc[3913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:38.234552 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:38.228112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48200 (local address 127.0.98.62:34889)
0926 09:27:38.228171 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:38.228173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:38.228181 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:27:38.228228 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:38.228229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:38.228237 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:38.228287 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:38.228771 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:38.229310 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:38.230045 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:38.230235 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:38.230295 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:38.232361 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:38.232362 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:38.232363 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:38.232370 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:38.233539 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:38.233887 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:38.233888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:38.233888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:38.233938 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:38.234098 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:38.234099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:38.234099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:38.234245 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:38.234321 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:38.234401 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 09:27:38.234462 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 09:27:38.234827 3958 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48200
I20260926 09:27:38.235217 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 4020
I20260926 09:27:38.239666 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 4154
I20260926 09:27:38.243690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 4288
I20260926 09:27:38.247763 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 3926
2026-09-26T09:27:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1865 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:27:38 dist-test-slave-dfzm krb5kdc[4429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:27:38 dist-test-slave-dfzm krb5kdc[4429](info): set up 2 sockets
Sep 26 09:27:38 dist-test-slave-dfzm krb5kdc[4429](info): commencing operation
krb5kdc: starting...
W20260926 09:27:39.309535 392 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:27:39 dist-test-slave-dfzm krb5kdc[4429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:27:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:27:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:27:39.426872 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43991
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:39.507355 4442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:39.507575 4442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:39.507596 4442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:39.508808 4442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:27:39.508854 4442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:39.508869 4442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:39.508882 4442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:27:39.508894 4442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:27:39.510452 4442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:39.510718 4442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:39.510952 4442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:39.515920 4450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:39.515910 4447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:39.515913 4448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:39.516227 4442 server_base.cc:1034] running on GCE node
I20260926 09:27:39.516511 4442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:39.516757 4442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:39.517896 4442 hybrid_clock.cc:648] HybridClock initialized: now 1790414859517882 us; error 29 us; skew 500 ppm
Sep 26 09:27:39 dist-test-slave-dfzm krb5kdc[4429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:39.520309 4442 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:27:39.520649 4442 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:39.520722 4442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:39.520872 4442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:39.522032 4442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "c4eaa4e2c85f4b13b679673c350222b3"
format_stamp: "Formatted at 2026-09-26 09:27:39 on dist-test-slave-dfzm"
server_key: "2c19f059cb8412ad53b62c76e9a740a3"
server_key_iv: "05dda057c49517ba5fc608303e5616a5"
server_key_version: "encryptionkey@0"
I20260926 09:27:39.522396 4442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "c4eaa4e2c85f4b13b679673c350222b3"
format_stamp: "Formatted at 2026-09-26 09:27:39 on dist-test-slave-dfzm"
server_key: "2c19f059cb8412ad53b62c76e9a740a3"
server_key_iv: "05dda057c49517ba5fc608303e5616a5"
server_key_version: "encryptionkey@0"
I20260926 09:27:39.523684 4442 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:27:39.524425 4456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:39.524597 4442 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:39.524703 4442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "c4eaa4e2c85f4b13b679673c350222b3"
format_stamp: "Formatted at 2026-09-26 09:27:39 on dist-test-slave-dfzm"
server_key: "2c19f059cb8412ad53b62c76e9a740a3"
server_key_iv: "05dda057c49517ba5fc608303e5616a5"
server_key_version: "encryptionkey@0"
I20260926 09:27:39.524780 4442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:39.542097 4442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:39.544014 4442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:39.544180 4442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:39.548161 4442 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43991
I20260926 09:27:39.548189 4508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43991 every 8 connection(s)
I20260926 09:27:39.548533 4442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:27:39.549782 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:39.551975 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 4442
I20260926 09:27:39.552089 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:27:39.552274 392 external_mini_cluster.cc:1428] Setting key 0633da73e1ae3887799c065cc38d6a89
I20260926 09:27:39.552315 4509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4eaa4e2c85f4b13b679673c350222b3: Bootstrap starting.
I20260926 09:27:39.553051 4509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4eaa4e2c85f4b13b679673c350222b3: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:39.553360 4509 log.cc:824] T 00000000000000000000000000000000 P c4eaa4e2c85f4b13b679673c350222b3: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:39.554039 4509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4eaa4e2c85f4b13b679673c350222b3: No bootstrap required, opened a new log
I20260926 09:27:39.555819 4509 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4eaa4e2c85f4b13b679673c350222b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4eaa4e2c85f4b13b679673c350222b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43991 } }
I20260926 09:27:39.555949 4509 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4eaa4e2c85f4b13b679673c350222b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:39.555971 4509 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4eaa4e2c85f4b13b679673c350222b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4eaa4e2c85f4b13b679673c350222b3, State: Initialized, Role: FOLLOWER
I20260926 09:27:39.556104 4509 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4eaa4e2c85f4b13b679673c350222b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4eaa4e2c85f4b13b679673c350222b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43991 } }
I20260926 09:27:39.556201 4509 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4eaa4e2c85f4b13b679673c350222b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:27:39.556247 4509 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4eaa4e2c85f4b13b679673c350222b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:27:39.556299 4509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4eaa4e2c85f4b13b679673c350222b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:39.556826 4509 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4eaa4e2c85f4b13b679673c350222b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4eaa4e2c85f4b13b679673c350222b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43991 } }
I20260926 09:27:39.556943 4509 leader_election.cc:302] T 00000000000000000000000000000000 P c4eaa4e2c85f4b13b679673c350222b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4eaa4e2c85f4b13b679673c350222b3; no voters:
I20260926 09:27:39.557142 4509 leader_election.cc:288] T 00000000000000000000000000000000 P c4eaa4e2c85f4b13b679673c350222b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:27:39.557185 4514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4eaa4e2c85f4b13b679673c350222b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:39.557331 4514 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4eaa4e2c85f4b13b679673c350222b3 [term 1 LEADER]: Becoming Leader. State: Replica: c4eaa4e2c85f4b13b679673c350222b3, State: Running, Role: LEADER
I20260926 09:27:39.557389 4509 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4eaa4e2c85f4b13b679673c350222b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:27:39.557420 4514 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4eaa4e2c85f4b13b679673c350222b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4eaa4e2c85f4b13b679673c350222b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43991 } }
I20260926 09:27:39.558641 4515 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4eaa4e2c85f4b13b679673c350222b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4eaa4e2c85f4b13b679673c350222b3. Latest consensus state: current_term: 1 leader_uuid: "c4eaa4e2c85f4b13b679673c350222b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4eaa4e2c85f4b13b679673c350222b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43991 } } }
I20260926 09:27:39.558729 4515 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4eaa4e2c85f4b13b679673c350222b3 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:39.558808 4516 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4eaa4e2c85f4b13b679673c350222b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4eaa4e2c85f4b13b679673c350222b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4eaa4e2c85f4b13b679673c350222b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43991 } } }
I20260926 09:27:39.558936 4516 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4eaa4e2c85f4b13b679673c350222b3 [sys.catalog]: This master's current role is: LEADER
Sep 26 09:27:39 dist-test-slave-dfzm krb5kdc[4429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:39.559058 4523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:27:39.559585 4523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:27:39.561754 4523 catalog_manager.cc:1347] Generated new cluster ID: 10d6371992a7494ab35ba118688668e0
I20260926 09:27:39.561820 4523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:27:39.563264 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:39.552812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42142 (local address 127.0.98.62:43991)
0926 09:27:39.552974 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:39.552984 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:39.553020 (+ 36us) server_negotiation.cc:407] Connection header received
0926 09:27:39.553547 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:39.553554 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:39.553646 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:39.553816 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:39.554484 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:39.555359 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:39.556195 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:39.556688 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:39.556809 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:39.559566 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:39.559569 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:39.559570 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:39.559582 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:39.561724 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:39.562113 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:39.562116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:39.562116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:39.562211 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:39.562426 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:39.562427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:39.562427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:39.562670 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:39.562845 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:39.562910 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 09:27:39.563044 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260926 09:27:39.586952 4523 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:27:39.587538 4523 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:27:39.594967 4523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4eaa4e2c85f4b13b679673c350222b3: Generated new TSK 0
I20260926 09:27:39.595176 4523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:27:39.628749 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43991
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:39.711248 4535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:39.711460 4535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:39.711484 4535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:39.712690 4535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:39.712733 4535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:39.712760 4535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:27:39.714337 4535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:39.714524 4535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:39.714692 4535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:39.719736 4543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:39.719728 4540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:39.719718 4541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:39.719977 4535 server_base.cc:1034] running on GCE node
I20260926 09:27:39.720188 4535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:39.720409 4535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:39.721551 4535 hybrid_clock.cc:648] HybridClock initialized: now 1790414859721539 us; error 29 us; skew 500 ppm
Sep 26 09:27:39 dist-test-slave-dfzm krb5kdc[4429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:39.724244 4535 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:27:39.724620 4535 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:39.724668 4535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:39.724830 4535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:39.725970 4535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "78b13da31fe047cfb1cb06615c4912a2"
format_stamp: "Formatted at 2026-09-26 09:27:39 on dist-test-slave-dfzm"
server_key: "cd4799fddd27d548ce3d2d0dca97d5c1"
server_key_iv: "466745ce6cb09430ded11b11be3f0b94"
server_key_version: "encryptionkey@0"
I20260926 09:27:39.726323 4535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "78b13da31fe047cfb1cb06615c4912a2"
format_stamp: "Formatted at 2026-09-26 09:27:39 on dist-test-slave-dfzm"
server_key: "cd4799fddd27d548ce3d2d0dca97d5c1"
server_key_iv: "466745ce6cb09430ded11b11be3f0b94"
server_key_version: "encryptionkey@0"
I20260926 09:27:39.727697 4535 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:27:39.728523 4549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:39.728718 4535 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:39.728826 4535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "78b13da31fe047cfb1cb06615c4912a2"
format_stamp: "Formatted at 2026-09-26 09:27:39 on dist-test-slave-dfzm"
server_key: "cd4799fddd27d548ce3d2d0dca97d5c1"
server_key_iv: "466745ce6cb09430ded11b11be3f0b94"
server_key_version: "encryptionkey@0"
I20260926 09:27:39.728905 4535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:39.744297 4535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:39.746765 4535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:39.746886 4535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:39.747117 4535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:39.747478 4535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:39.747535 4535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:39.747577 4535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:39.747619 4535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:39.753418 4535 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43335
I20260926 09:27:39.753472 4662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43335 every 8 connection(s)
I20260926 09:27:39.753818 4535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 09:27:39 dist-test-slave-dfzm krb5kdc[4429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:39.762780 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:39.754559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36107 (local address 127.0.98.62:43991)
0926 09:27:39.754640 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:39.754643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:39.754820 (+ 177us) server_negotiation.cc:407] Connection header received
0926 09:27:39.755291 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:39.755292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:39.755301 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:39.755355 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:39.756298 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:39.756788 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:39.757878 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:39.758080 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:39.758171 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:39.760395 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:39.760396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:39.760396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:39.760403 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:39.761562 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:39.762000 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:39.762001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:39.762001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:39.762054 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:39.762264 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:39.762264 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:39.762264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:39.762368 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:39.762418 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:39.762648 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 09:27:39.762714 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 09:27:39.762975 4665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:39.754518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43991 (local address 127.0.98.1:36107)
0926 09:27:39.754696 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:39.754733 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:39.755202 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:39.755411 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:39.755414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:39.755671 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:39.756209 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:39.756214 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:39.756872 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:39.756873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:39.757799 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:39.757801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:39.758198 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:39.758199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:39.758329 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:39.758855 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:39.758863 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:39.760307 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:39.761713 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:39.761719 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:39.761721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:39.761936 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:39.762112 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:39.762112 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:39.762113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:39.762209 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:39.762421 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:39.762424 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:39.762540 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 09:27:39.762621 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 09:27:39.762736 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
I20260926 09:27:39.763581 4663 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43991
I20260926 09:27:39.763695 4663 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:39.763828 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 4535
I20260926 09:27:39.763898 4663 heartbeater.cc:511] Master 127.0.98.62:43991 requested a full tablet report, sending...
I20260926 09:27:39.763947 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:27:39.764146 392 external_mini_cluster.cc:1428] Setting key e76db3d7f70dff62e4170727e0bdffeb
I20260926 09:27:39.764468 4473 ts_manager.cc:194] Registered new tserver with Master: 78b13da31fe047cfb1cb06615c4912a2 (127.0.98.1:43335)
I20260926 09:27:39.765188 4473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36107
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:27:39.789032 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43991
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:39.870323 4668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:39.870566 4668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:39.870609 4668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:39.871814 4668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:39.871882 4668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:39.871932 4668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:27:39.873538 4668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:39.873840 4668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:39.874071 4668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:39.879160 4676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:39.879168 4674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:39.879359 4668 server_base.cc:1034] running on GCE node
W20260926 09:27:39.879182 4673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:39.879631 4668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:39.879887 4668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:39.881052 4668 hybrid_clock.cc:648] HybridClock initialized: now 1790414859881040 us; error 53 us; skew 500 ppm
Sep 26 09:27:39 dist-test-slave-dfzm krb5kdc[4429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:39.883504 4668 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:27:39.883859 4668 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:39.883934 4668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:39.884080 4668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:39.885120 4668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "8149e745135e40ca8cee95213bf820c7"
format_stamp: "Formatted at 2026-09-26 09:27:39 on dist-test-slave-dfzm"
server_key: "7d15996309b9046134714c2cd290866c"
server_key_iv: "bcf9ede3de78b71d364f49cf78babe3a"
server_key_version: "encryptionkey@0"
I20260926 09:27:39.885489 4668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8149e745135e40ca8cee95213bf820c7"
format_stamp: "Formatted at 2026-09-26 09:27:39 on dist-test-slave-dfzm"
server_key: "7d15996309b9046134714c2cd290866c"
server_key_iv: "bcf9ede3de78b71d364f49cf78babe3a"
server_key_version: "encryptionkey@0"
I20260926 09:27:39.886762 4668 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:27:39.887476 4682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:39.887643 4668 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:27:39.887745 4668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "8149e745135e40ca8cee95213bf820c7"
format_stamp: "Formatted at 2026-09-26 09:27:39 on dist-test-slave-dfzm"
server_key: "7d15996309b9046134714c2cd290866c"
server_key_iv: "bcf9ede3de78b71d364f49cf78babe3a"
server_key_version: "encryptionkey@0"
I20260926 09:27:39.887823 4668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:39.904060 4668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:39.906273 4668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:39.906442 4668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:39.906690 4668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:39.907008 4668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:39.907065 4668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:39.907106 4668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:39.907136 4668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:39.913657 4668 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39367
I20260926 09:27:39.913708 4795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39367 every 8 connection(s)
I20260926 09:27:39.913995 4668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:27:39 dist-test-slave-dfzm krb5kdc[4429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:39.923064 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:39.914767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44819 (local address 127.0.98.62:43991)
0926 09:27:39.914834 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:39.914836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:39.915054 (+ 218us) server_negotiation.cc:407] Connection header received
0926 09:27:39.915537 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:39.915538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:39.915547 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:39.915607 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:39.916519 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:39.917012 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:39.918082 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:39.918257 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:39.918316 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:39.920282 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:39.920283 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:39.920283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:39.920290 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:39.921737 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:39.922138 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:39.922139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:39.922140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:39.922194 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:39.922463 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:39.922465 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:39.922465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:39.922600 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:39.922690 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:39.922925 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 09:27:39.922996 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:27:39.923209 4798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:39.914759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43991 (local address 127.0.98.2:44819)
0926 09:27:39.914928 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:39.914970 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:39.915454 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:39.915660 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:39.915664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:39.915923 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:39.916431 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:39.916436 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:39.917095 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:39.917095 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:39.918009 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:39.918011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:39.918329 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:39.918329 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:39.918437 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:39.918885 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:39.918893 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:39.920200 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:39.921821 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:39.921826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:39.921828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:39.922056 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:39.922257 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:39.922257 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:39.922257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:39.922377 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:39.922688 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:39.922692 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:39.922804 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 09:27:39.922886 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 09:27:39.923002 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":102,"threads_started":1}
I20260926 09:27:39.923717 4796 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43991
I20260926 09:27:39.923821 4796 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:39.924022 4796 heartbeater.cc:511] Master 127.0.98.62:43991 requested a full tablet report, sending...
I20260926 09:27:39.924048 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 4668
I20260926 09:27:39.924181 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:27:39.924394 392 external_mini_cluster.cc:1428] Setting key 573fb34923932e4b1e5b6606f8baac46
I20260926 09:27:39.924506 4473 ts_manager.cc:194] Registered new tserver with Master: 8149e745135e40ca8cee95213bf820c7 (127.0.98.2:39367)
I20260926 09:27:39.924917 4473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44819
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:27:39.948524 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43991
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:40.029012 4801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:40.029219 4801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:40.029244 4801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:40.030424 4801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:40.030467 4801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:40.030495 4801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:27:40.031989 4801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:40.032220 4801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:40.032416 4801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:40.037451 4809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:40.037454 4806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:40.037509 4807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:40.037915 4801 server_base.cc:1034] running on GCE node
I20260926 09:27:40.038151 4801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:40.038429 4801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:40.039598 4801 hybrid_clock.cc:648] HybridClock initialized: now 1790414860039586 us; error 27 us; skew 500 ppm
Sep 26 09:27:40 dist-test-slave-dfzm krb5kdc[4429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:40.042279 4801 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:27:40.042665 4801 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:40.042716 4801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:40.042891 4801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:40.043947 4801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "9f74b9868e374e10bc7999d4fe3a73f5"
format_stamp: "Formatted at 2026-09-26 09:27:40 on dist-test-slave-dfzm"
server_key: "04cad8904155a7c14f359fac76266d3a"
server_key_iv: "3ff3d5c6b33658c248fd6d858ebf7b9d"
server_key_version: "encryptionkey@0"
I20260926 09:27:40.044294 4801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9f74b9868e374e10bc7999d4fe3a73f5"
format_stamp: "Formatted at 2026-09-26 09:27:40 on dist-test-slave-dfzm"
server_key: "04cad8904155a7c14f359fac76266d3a"
server_key_iv: "3ff3d5c6b33658c248fd6d858ebf7b9d"
server_key_version: "encryptionkey@0"
I20260926 09:27:40.045694 4801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:27:40.046471 4815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:40.046711 4801 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:40.046783 4801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "9f74b9868e374e10bc7999d4fe3a73f5"
format_stamp: "Formatted at 2026-09-26 09:27:40 on dist-test-slave-dfzm"
server_key: "04cad8904155a7c14f359fac76266d3a"
server_key_iv: "3ff3d5c6b33658c248fd6d858ebf7b9d"
server_key_version: "encryptionkey@0"
I20260926 09:27:40.046880 4801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:40.059036 4801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:40.061014 4801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:40.061172 4801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:40.061383 4801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:40.061740 4801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:40.061775 4801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:40.061798 4801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:40.061841 4801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:40.067772 4801 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34331
I20260926 09:27:40.067807 4928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34331 every 8 connection(s)
I20260926 09:27:40.068075 4801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:27:40.073431 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 4801
I20260926 09:27:40.073516 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:27:40.073722 392 external_mini_cluster.cc:1428] Setting key 2ee0f2ba6b7f8deb651fb5865c0c4710
Sep 26 09:27:40 dist-test-slave-dfzm krb5kdc[4429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:40.077278 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:40.068751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33615 (local address 127.0.98.62:43991)
0926 09:27:40.068793 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:40.068797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:40.069010 (+ 213us) server_negotiation.cc:407] Connection header received
0926 09:27:40.069459 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:40.069460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:40.069469 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:40.069525 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:40.070486 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.070969 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.071909 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.072086 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.072135 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:40.074357 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:40.074359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:40.074359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:40.074368 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:40.075858 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:40.076284 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:40.076286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:40.076286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:40.076352 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:40.076619 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:40.076619 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:40.076620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:40.076741 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:40.076812 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:40.077101 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 09:27:40.077181 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 09:27:40.077486 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:40.068717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43991 (local address 127.0.98.3:33615)
0926 09:27:40.068880 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:40.068921 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:40.069381 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:40.069579 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:40.069633 (+ 54us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:40.069905 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:40.070403 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:40.070408 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.071047 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.071048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:40.071829 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:40.071831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.072155 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.072156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:40.072250 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:40.072688 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:40.072696 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:40.074257 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:40.075964 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:40.075971 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:40.075973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:40.076206 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:40.076422 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:40.076423 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:40.076423 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:40.076542 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:40.076822 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:40.076829 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:40.076968 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 09:27:40.077080 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 09:27:40.077244 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260926 09:27:40.077955 4929 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43991
I20260926 09:27:40.078037 4929 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:40.078184 4929 heartbeater.cc:511] Master 127.0.98.62:43991 requested a full tablet report, sending...
I20260926 09:27:40.078501 4473 ts_manager.cc:194] Registered new tserver with Master: 9f74b9868e374e10bc7999d4fe3a73f5 (127.0.98.3:34331)
I20260926 09:27:40.078895 4473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33615
I20260926 09:27:40.085311 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:27:40.092461 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:40.086655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42156 (local address 127.0.98.62:43991)
0926 09:27:40.086700 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:40.086702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:40.086802 (+ 100us) server_negotiation.cc:407] Connection header received
0926 09:27:40.086845 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:40.086846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:40.086854 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:40.086905 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:40.087433 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.087894 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.088573 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.088742 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.088791 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:40.089986 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:40.089987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:40.089987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:40.089995 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:40.091480 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:40.091800 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:40.091801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:40.091801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:40.091862 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:40.092026 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:40.092026 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:40.092027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:40.092166 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:40.092240 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:40.092340 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 09:27:40.092398 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:27:40.093977 4473 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:27:40.095665 4473 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:27:40.102025 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:40.097971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34331 (local address 127.0.0.1:35818)
0926 09:27:40.098238 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:40.098249 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:40.098300 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:40.098576 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:40.098579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:40.098581 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:40.098652 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:40.098653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.099694 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.099695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:40.100786 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:40.100788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.101681 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.101682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:40.101772 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:40.101784 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 09:27:40.101887 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 09:27:40.101944 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":79,"threads_started":1}
I20260926 09:27:40.102036 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:40.098072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35818 (local address 127.0.98.3:34331)
0926 09:27:40.098394 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:40.098399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:40.098408 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:27:40.098451 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:40.098453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:40.098465 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:40.098549 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:40.098781 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.099594 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.100894 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.101562 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.101671 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:40.101736 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:40.101905 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 09:27:40.101945 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":72,"threads_started":1}
I20260926 09:27:40.102445 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:40.097965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39367 (local address 127.0.0.1:43578)
0926 09:27:40.098127 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:40.098169 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:40.098232 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:40.098779 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:40.098780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:40.098783 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:40.098851 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:40.098853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.100204 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.100205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:40.101276 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:40.101279 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.102244 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.102245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:40.102301 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:40.102305 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:27:40.102333 (+ 28us) client_negotiation.cc:240] Negotiation successful
0926 09:27:40.102369 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":15,"thread_start_us":55,"threads_started":1}
I20260926 09:27:40.102670 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:40.098160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43335 (local address 127.0.0.1:56884)
0926 09:27:40.098280 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:40.098290 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:40.098332 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:40.098795 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:40.098796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:40.098797 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:40.098939 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:40.098940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.100213 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.100214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:40.101106 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:40.101108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.102440 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.102441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:40.102507 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:40.102512 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:27:40.102570 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 09:27:40.102613 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":3,"thread_start_us":82,"threads_started":1}
I20260926 09:27:40.102789 4863 tablet_service.cc:1467] Processing CreateTablet for tablet afa88d9cd32542a1a2a5a6fead9b1dc6 (DEFAULT_TABLE table=test-table [id=00e6fb48d74745bba5617d95867bc2b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:40.102944 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:40.098072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43578 (local address 127.0.98.2:39367)
0926 09:27:40.098613 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:40.098617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:40.098626 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:27:40.098664 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:40.098665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:40.098679 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:40.098828 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:40.098979 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.100079 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.101369 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.102124 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.102619 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:40.102704 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:40.102805 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 09:27:40.102848 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":79,"threads_started":1}
I20260926 09:27:40.102962 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:40.098398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56884 (local address 127.0.98.1:43335)
0926 09:27:40.098554 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:40.098559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:40.098571 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:27:40.098612 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:40.098614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:40.098631 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:40.098717 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:40.099015 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.100111 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.101193 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.101982 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.102056 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:40.102797 (+ 741us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:40.102823 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 09:27:40.102862 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260926 09:27:40.103188 4863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa88d9cd32542a1a2a5a6fead9b1dc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:40.103541 4730 tablet_service.cc:1467] Processing CreateTablet for tablet afa88d9cd32542a1a2a5a6fead9b1dc6 (DEFAULT_TABLE table=test-table [id=00e6fb48d74745bba5617d95867bc2b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:40.103541 4597 tablet_service.cc:1467] Processing CreateTablet for tablet afa88d9cd32542a1a2a5a6fead9b1dc6 (DEFAULT_TABLE table=test-table [id=00e6fb48d74745bba5617d95867bc2b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:40.103860 4597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa88d9cd32542a1a2a5a6fead9b1dc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:40.103868 4730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa88d9cd32542a1a2a5a6fead9b1dc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:40.106197 4946 tablet_bootstrap.cc:492] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 9f74b9868e374e10bc7999d4fe3a73f5: Bootstrap starting.
I20260926 09:27:40.106778 4946 tablet_bootstrap.cc:654] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 9f74b9868e374e10bc7999d4fe3a73f5: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:40.107087 4948 tablet_bootstrap.cc:492] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 8149e745135e40ca8cee95213bf820c7: Bootstrap starting.
I20260926 09:27:40.107416 4946 log.cc:824] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 9f74b9868e374e10bc7999d4fe3a73f5: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:40.107708 4948 tablet_bootstrap.cc:654] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 8149e745135e40ca8cee95213bf820c7: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:40.108147 4946 tablet_bootstrap.cc:492] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 9f74b9868e374e10bc7999d4fe3a73f5: No bootstrap required, opened a new log
I20260926 09:27:40.108233 4946 ts_tablet_manager.cc:1397] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 9f74b9868e374e10bc7999d4fe3a73f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:27:40.108217 4948 log.cc:824] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 8149e745135e40ca8cee95213bf820c7: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:40.108587 4947 tablet_bootstrap.cc:492] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2: Bootstrap starting.
I20260926 09:27:40.109095 4947 tablet_bootstrap.cc:654] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:40.109709 4947 log.cc:824] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:40.109936 4948 tablet_bootstrap.cc:492] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 8149e745135e40ca8cee95213bf820c7: No bootstrap required, opened a new log
I20260926 09:27:40.110002 4948 ts_tablet_manager.cc:1397] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 8149e745135e40ca8cee95213bf820c7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 09:27:40.110066 4946 raft_consensus.cc:348] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 9f74b9868e374e10bc7999d4fe3a73f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8149e745135e40ca8cee95213bf820c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39367 } } peers { permanent_uuid: "9f74b9868e374e10bc7999d4fe3a73f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34331 } } peers { permanent_uuid: "78b13da31fe047cfb1cb06615c4912a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43335 } }
I20260926 09:27:40.110242 4946 raft_consensus.cc:374] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 9f74b9868e374e10bc7999d4fe3a73f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:40.110309 4946 raft_consensus.cc:729] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 9f74b9868e374e10bc7999d4fe3a73f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f74b9868e374e10bc7999d4fe3a73f5, State: Initialized, Role: FOLLOWER
I20260926 09:27:40.110457 4946 consensus_queue.cc:260] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 9f74b9868e374e10bc7999d4fe3a73f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8149e745135e40ca8cee95213bf820c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39367 } } peers { permanent_uuid: "9f74b9868e374e10bc7999d4fe3a73f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34331 } } peers { permanent_uuid: "78b13da31fe047cfb1cb06615c4912a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43335 } }
I20260926 09:27:40.110591 4947 tablet_bootstrap.cc:492] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2: No bootstrap required, opened a new log
I20260926 09:27:40.110673 4947 ts_tablet_manager.cc:1397] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 09:27:40.110744 4946 ts_tablet_manager.cc:1428] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 9f74b9868e374e10bc7999d4fe3a73f5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 09:27:40.110874 4929 heartbeater.cc:503] Master 127.0.98.62:43991 was elected leader, sending a full tablet report...
I20260926 09:27:40.112028 4948 raft_consensus.cc:348] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 8149e745135e40ca8cee95213bf820c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8149e745135e40ca8cee95213bf820c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39367 } } peers { permanent_uuid: "9f74b9868e374e10bc7999d4fe3a73f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34331 } } peers { permanent_uuid: "78b13da31fe047cfb1cb06615c4912a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43335 } }
I20260926 09:27:40.112185 4948 raft_consensus.cc:374] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 8149e745135e40ca8cee95213bf820c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:40.112218 4948 raft_consensus.cc:729] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 8149e745135e40ca8cee95213bf820c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8149e745135e40ca8cee95213bf820c7, State: Initialized, Role: FOLLOWER
I20260926 09:27:40.112329 4948 consensus_queue.cc:260] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 8149e745135e40ca8cee95213bf820c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8149e745135e40ca8cee95213bf820c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39367 } } peers { permanent_uuid: "9f74b9868e374e10bc7999d4fe3a73f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34331 } } peers { permanent_uuid: "78b13da31fe047cfb1cb06615c4912a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43335 } }
I20260926 09:27:40.112612 4948 ts_tablet_manager.cc:1428] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 8149e745135e40ca8cee95213bf820c7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:27:40.112653 4796 heartbeater.cc:503] Master 127.0.98.62:43991 was elected leader, sending a full tablet report...
I20260926 09:27:40.112696 4947 raft_consensus.cc:348] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8149e745135e40ca8cee95213bf820c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39367 } } peers { permanent_uuid: "9f74b9868e374e10bc7999d4fe3a73f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34331 } } peers { permanent_uuid: "78b13da31fe047cfb1cb06615c4912a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43335 } }
I20260926 09:27:40.112839 4947 raft_consensus.cc:374] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:40.112890 4947 raft_consensus.cc:729] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b13da31fe047cfb1cb06615c4912a2, State: Initialized, Role: FOLLOWER
I20260926 09:27:40.112999 4947 consensus_queue.cc:260] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8149e745135e40ca8cee95213bf820c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39367 } } peers { permanent_uuid: "9f74b9868e374e10bc7999d4fe3a73f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34331 } } peers { permanent_uuid: "78b13da31fe047cfb1cb06615c4912a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43335 } }
I20260926 09:27:40.113266 4947 ts_tablet_manager.cc:1428] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 09:27:40.113375 4663 heartbeater.cc:503] Master 127.0.98.62:43991 was elected leader, sending a full tablet report...
W20260926 09:27:40.164575 4797 tablet.cc:2369] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 8149e745135e40ca8cee95213bf820c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:27:40.254529 4664 tablet.cc:2369] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:27:40.318599 4930 tablet.cc:2369] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 9f74b9868e374e10bc7999d4fe3a73f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:27:40.371227 4954 raft_consensus.cc:482] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:27:40.371382 4954 raft_consensus.cc:504] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8149e745135e40ca8cee95213bf820c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39367 } } peers { permanent_uuid: "9f74b9868e374e10bc7999d4fe3a73f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34331 } } peers { permanent_uuid: "78b13da31fe047cfb1cb06615c4912a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43335 } }
I20260926 09:27:40.371755 4954 leader_election.cc:288] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8149e745135e40ca8cee95213bf820c7 (127.0.98.2:39367), 9f74b9868e374e10bc7999d4fe3a73f5 (127.0.98.3:34331)
I20260926 09:27:40.374817 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:40.372032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36661 (local address 127.0.98.2:39367)
0926 09:27:40.372080 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:40.372088 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:40.372218 (+ 130us) server_negotiation.cc:407] Connection header received
0926 09:27:40.372279 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:40.372280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:40.372292 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:40.372353 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:40.372530 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.373007 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.374072 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.374447 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.374538 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:40.374548 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:40.374703 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 09:27:40.374742 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:27:40.374838 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:40.371973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39367 (local address 127.0.98.1:36661)
0926 09:27:40.372125 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:40.372138 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:40.372212 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:40.372372 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:40.372374 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:40.372375 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:40.372454 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:40.372456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.373098 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.373099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:40.373977 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:40.373979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.374539 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.374540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:40.374621 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:40.374625 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:27:40.374695 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 09:27:40.374741 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
I20260926 09:27:40.375174 4750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa88d9cd32542a1a2a5a6fead9b1dc6" candidate_uuid: "78b13da31fe047cfb1cb06615c4912a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8149e745135e40ca8cee95213bf820c7" is_pre_election: true
I20260926 09:27:40.375310 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:40.372001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34331 (local address 127.0.98.1:46367)
0926 09:27:40.372129 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:40.372140 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:40.372230 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:40.372430 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:40.372431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:40.372433 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:40.372527 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:40.372528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.373178 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.373180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:40.374338 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:40.374340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.375029 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.375030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:40.375109 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:40.375115 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 09:27:40.375189 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 09:27:40.375226 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":58,"threads_started":1}
I20260926 09:27:40.375295 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:40.372080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46367 (local address 127.0.98.3:34331)
0926 09:27:40.372139 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:40.372141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:40.372220 (+ 79us) server_negotiation.cc:407] Connection header received
0926 09:27:40.372331 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:40.372332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:40.372349 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:40.372426 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:40.372602 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.373096 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.374442 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.374906 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.374978 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:40.374988 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:40.375196 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 09:27:40.375232 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 09:27:40.375379 4750 raft_consensus.cc:2454] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 8149e745135e40ca8cee95213bf820c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b13da31fe047cfb1cb06615c4912a2 in term 0.
I20260926 09:27:40.375663 4552 leader_election.cc:302] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b13da31fe047cfb1cb06615c4912a2, 8149e745135e40ca8cee95213bf820c7; no voters:
I20260926 09:27:40.375846 4883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa88d9cd32542a1a2a5a6fead9b1dc6" candidate_uuid: "78b13da31fe047cfb1cb06615c4912a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f74b9868e374e10bc7999d4fe3a73f5" is_pre_election: true
I20260926 09:27:40.375877 4954 raft_consensus.cc:2781] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:27:40.375976 4954 raft_consensus.cc:482] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:27:40.376003 4954 raft_consensus.cc:3037] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:40.375996 4883 raft_consensus.cc:2454] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 9f74b9868e374e10bc7999d4fe3a73f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b13da31fe047cfb1cb06615c4912a2 in term 0.
I20260926 09:27:40.380857 4954 raft_consensus.cc:504] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8149e745135e40ca8cee95213bf820c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39367 } } peers { permanent_uuid: "9f74b9868e374e10bc7999d4fe3a73f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34331 } } peers { permanent_uuid: "78b13da31fe047cfb1cb06615c4912a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43335 } }
I20260926 09:27:40.381181 4954 leader_election.cc:288] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2 [CANDIDATE]: Term 1 election: Requested vote from peers 8149e745135e40ca8cee95213bf820c7 (127.0.98.2:39367), 9f74b9868e374e10bc7999d4fe3a73f5 (127.0.98.3:34331)
I20260926 09:27:40.382017 4883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa88d9cd32542a1a2a5a6fead9b1dc6" candidate_uuid: "78b13da31fe047cfb1cb06615c4912a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f74b9868e374e10bc7999d4fe3a73f5"
I20260926 09:27:40.382150 4883 raft_consensus.cc:3037] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 9f74b9868e374e10bc7999d4fe3a73f5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:40.383270 4883 raft_consensus.cc:2454] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 9f74b9868e374e10bc7999d4fe3a73f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b13da31fe047cfb1cb06615c4912a2 in term 1.
I20260926 09:27:40.383489 4551 leader_election.cc:302] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b13da31fe047cfb1cb06615c4912a2, 9f74b9868e374e10bc7999d4fe3a73f5; no voters:
I20260926 09:27:40.383596 4954 raft_consensus.cc:2781] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:40.383798 4954 raft_consensus.cc:686] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2 [term 1 LEADER]: Becoming Leader. State: Replica: 78b13da31fe047cfb1cb06615c4912a2, State: Running, Role: LEADER
I20260926 09:27:40.383934 4954 consensus_queue.cc:237] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8149e745135e40ca8cee95213bf820c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39367 } } peers { permanent_uuid: "9f74b9868e374e10bc7999d4fe3a73f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34331 } } peers { permanent_uuid: "78b13da31fe047cfb1cb06615c4912a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43335 } }
I20260926 09:27:40.384804 4473 catalog_manager.cc:5495] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b13da31fe047cfb1cb06615c4912a2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "78b13da31fe047cfb1cb06615c4912a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8149e745135e40ca8cee95213bf820c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f74b9868e374e10bc7999d4fe3a73f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b13da31fe047cfb1cb06615c4912a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43335 } health_report { overall_health: HEALTHY } } }
I20260926 09:27:40.385061 4750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa88d9cd32542a1a2a5a6fead9b1dc6" candidate_uuid: "78b13da31fe047cfb1cb06615c4912a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8149e745135e40ca8cee95213bf820c7"
I20260926 09:27:40.385160 4750 raft_consensus.cc:3037] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 8149e745135e40ca8cee95213bf820c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:40.386226 4750 raft_consensus.cc:2454] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 8149e745135e40ca8cee95213bf820c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b13da31fe047cfb1cb06615c4912a2 in term 1.
I20260926 09:27:40.454393 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:40.451915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56890 (local address 127.0.98.1:43335)
0926 09:27:40.451977 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:40.451979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:40.451987 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:27:40.452027 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:40.452028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:40.452060 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:40.452144 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:27:40.452281 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.452756 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.453551 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:40.453777 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:40.453857 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:40.454041 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:27:40.454160 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:27:40.454250 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:40.454289 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 09:27:40.454310 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 09:27:40.456243 4883 raft_consensus.cc:1261] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 9f74b9868e374e10bc7999d4fe3a73f5 [term 1 FOLLOWER]: Refusing update from remote peer 78b13da31fe047cfb1cb06615c4912a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:40.456243 4750 raft_consensus.cc:1261] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 8149e745135e40ca8cee95213bf820c7 [term 1 FOLLOWER]: Refusing update from remote peer 78b13da31fe047cfb1cb06615c4912a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:40.456591 4957 consensus_queue.cc:1035] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8149e745135e40ca8cee95213bf820c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:40.456822 4954 consensus_queue.cc:1035] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 78b13da31fe047cfb1cb06615c4912a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f74b9868e374e10bc7999d4fe3a73f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:40.459957 4962 mvcc.cc:205] Tried to move back new op lower bound from 7333539268425961472 to 7333539268133249024. Current Snapshot: MvccSnapshot[applied={T|T < 7333539268425961472}]
I20260926 09:27:40.462177 4473 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:40.462284 4473 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:40.462992 4473 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4eaa4e2c85f4b13b679673c350222b3: Sending DeleteTablet for 3 replicas of tablet afa88d9cd32542a1a2a5a6fead9b1dc6
I20260926 09:27:40.463347 4863 tablet_service.cc:1514] Processing DeleteTablet for tablet afa88d9cd32542a1a2a5a6fead9b1dc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35818
I20260926 09:27:40.463346 4730 tablet_service.cc:1514] Processing DeleteTablet for tablet afa88d9cd32542a1a2a5a6fead9b1dc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43578
I20260926 09:27:40.463594 4971 tablet_replica.cc:330] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 8149e745135e40ca8cee95213bf820c7: stopping tablet replica
I20260926 09:27:40.463693 4971 raft_consensus.cc:2229] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 8149e745135e40ca8cee95213bf820c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:40.463742 4970 tablet_replica.cc:330] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 9f74b9868e374e10bc7999d4fe3a73f5: stopping tablet replica
I20260926 09:27:40.463727 4597 tablet_service.cc:1514] Processing DeleteTablet for tablet afa88d9cd32542a1a2a5a6fead9b1dc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56884
I20260926 09:27:40.463804 4971 raft_consensus.cc:2258] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 8149e745135e40ca8cee95213bf820c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:40.463840 4970 raft_consensus.cc:2229] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 9f74b9868e374e10bc7999d4fe3a73f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:40.464120 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 4535
I20260926 09:27:40.464249 4971 ts_tablet_manager.cc:1905] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 8149e745135e40ca8cee95213bf820c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:40.465375 4970 raft_consensus.cc:2258] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 9f74b9868e374e10bc7999d4fe3a73f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:40.465869 4970 ts_tablet_manager.cc:1905] T afa88d9cd32542a1a2a5a6fead9b1dc6 P 9f74b9868e374e10bc7999d4fe3a73f5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 09:27:40.470669 4459 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43335 (error 108)
I20260926 09:27:40.470878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 4668
I20260926 09:27:40.471714 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:40.471579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43335 (local address 127.0.0.1:56906)
0926 09:27:40.471637 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:40.471683 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 09:27:40.471858 4459 catalog_manager.cc:4584] TS 78b13da31fe047cfb1cb06615c4912a2 (127.0.98.1:43335): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet afa88d9cd32542a1a2a5a6fead9b1dc6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43335: connect: Connection refused (error 111)
I20260926 09:27:40.524535 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:40.524361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43335 (local address 127.0.0.1:56912)
0926 09:27:40.524445 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:40.524494 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 09:27:40.524927 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:40.524843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43335 (local address 127.0.0.1:56918)
0926 09:27:40.524887 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:40.524912 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 09:27:40.543897 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 4801
I20260926 09:27:40.543951 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:40.543783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39367 (local address 127.0.0.1:43594)
0926 09:27:40.543842 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:40.543886 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 09:27:40.548666 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:40.548540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34331 (local address 127.0.0.1:35820)
0926 09:27:40.548593 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:40.548623 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:34331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 09:27:40.548693 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 4442
2026-09-26T09:27:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2323 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T09:27:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:27:40Z Disabled control of system clock
I20260926 09:27:40.599516 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39857
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 09:27:40.689689 4977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:40.689949 4977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:40.689987 4977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:40.691211 4977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:27:40.691285 4977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:40.691309 4977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:40.691342 4977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:27:40.691368 4977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:27:40.692768 4977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39857
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:40.693068 4977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:40.693378 4977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:40.698326 4985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:40.698328 4982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:40.698302 4983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:40.698647 4977 server_base.cc:1034] running on GCE node
I20260926 09:27:40.698880 4977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:40.699219 4977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:40.700376 4977 hybrid_clock.cc:648] HybridClock initialized: now 1790414860700363 us; error 28 us; skew 500 ppm
I20260926 09:27:40.701813 4977 webserver.cc:460] Webserver started at http://127.0.98.62:35721/ using document root <none> and password file <none>
I20260926 09:27:40.702068 4977 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:40.702118 4977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:40.702279 4977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:40.703419 4977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "9e546dde2b1240bb921c146f099b9972"
format_stamp: "Formatted at 2026-09-26 09:27:40 on dist-test-slave-dfzm"
server_key: "a49fd8fd995f24910867a1e265415f36"
server_key_iv: "e0c20b2148fcd8fa86476f5202c8d9df"
server_key_version: "encryptionkey@0"
I20260926 09:27:40.703791 4977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "9e546dde2b1240bb921c146f099b9972"
format_stamp: "Formatted at 2026-09-26 09:27:40 on dist-test-slave-dfzm"
server_key: "a49fd8fd995f24910867a1e265415f36"
server_key_iv: "e0c20b2148fcd8fa86476f5202c8d9df"
server_key_version: "encryptionkey@0"
I20260926 09:27:40.705091 4977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:27:40.705848 4991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:40.706013 4977 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:40.706115 4977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "9e546dde2b1240bb921c146f099b9972"
format_stamp: "Formatted at 2026-09-26 09:27:40 on dist-test-slave-dfzm"
server_key: "a49fd8fd995f24910867a1e265415f36"
server_key_iv: "e0c20b2148fcd8fa86476f5202c8d9df"
server_key_version: "encryptionkey@0"
I20260926 09:27:40.706188 4977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:40.709473 4977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:40.709779 4977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:40.709937 4977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:40.713661 4977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39857
I20260926 09:27:40.713711 5043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39857 every 8 connection(s)
I20260926 09:27:40.714023 4977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:27:40.714671 5044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:40.717387 5044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e546dde2b1240bb921c146f099b9972: Bootstrap starting.
I20260926 09:27:40.718014 5044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e546dde2b1240bb921c146f099b9972: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:40.718331 5044 log.cc:824] T 00000000000000000000000000000000 P 9e546dde2b1240bb921c146f099b9972: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:40.719004 5044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e546dde2b1240bb921c146f099b9972: No bootstrap required, opened a new log
I20260926 09:27:40.721019 5044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e546dde2b1240bb921c146f099b9972 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e546dde2b1240bb921c146f099b9972" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39857 } }
I20260926 09:27:40.721159 5044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e546dde2b1240bb921c146f099b9972 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:40.721181 5044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e546dde2b1240bb921c146f099b9972 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e546dde2b1240bb921c146f099b9972, State: Initialized, Role: FOLLOWER
I20260926 09:27:40.721350 5044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e546dde2b1240bb921c146f099b9972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e546dde2b1240bb921c146f099b9972" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39857 } }
I20260926 09:27:40.721413 5044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e546dde2b1240bb921c146f099b9972 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:27:40.721441 5044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e546dde2b1240bb921c146f099b9972 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:27:40.721490 5044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e546dde2b1240bb921c146f099b9972 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:40.724252 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 4977
I20260926 09:27:40.724335 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:27:40.724470 392 external_mini_cluster.cc:1428] Setting key 8eb5f2d7b3750ebb224d8bc84f6b751c
I20260926 09:27:40.726616 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:40.724948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59066 (local address 127.0.98.62:39857)
0926 09:27:40.725210 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:40.725218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:40.725243 (+ 25us) server_negotiation.cc:407] Connection header received
0926 09:27:40.725340 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:40.725346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:40.725381 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:40.725527 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:40.726160 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:40.726164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:40.726164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:27:40.726168 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:40.726226 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:27:40.726237 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:40.726315 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:40.726386 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 09:27:40.726449 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":116,"threads_started":1}
I20260926 09:27:40.786386 5044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e546dde2b1240bb921c146f099b9972 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e546dde2b1240bb921c146f099b9972" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39857 } }
I20260926 09:27:40.786597 5044 leader_election.cc:302] T 00000000000000000000000000000000 P 9e546dde2b1240bb921c146f099b9972 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e546dde2b1240bb921c146f099b9972; no voters:
I20260926 09:27:40.786847 5044 leader_election.cc:288] T 00000000000000000000000000000000 P 9e546dde2b1240bb921c146f099b9972 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:27:40.786967 5049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e546dde2b1240bb921c146f099b9972 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:40.787160 5049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e546dde2b1240bb921c146f099b9972 [term 1 LEADER]: Becoming Leader. State: Replica: 9e546dde2b1240bb921c146f099b9972, State: Running, Role: LEADER
I20260926 09:27:40.787207 5044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e546dde2b1240bb921c146f099b9972 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:27:40.787261 5049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e546dde2b1240bb921c146f099b9972 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e546dde2b1240bb921c146f099b9972" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39857 } }
I20260926 09:27:40.787683 5050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e546dde2b1240bb921c146f099b9972 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e546dde2b1240bb921c146f099b9972" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e546dde2b1240bb921c146f099b9972" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39857 } } }
I20260926 09:27:40.787787 5050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e546dde2b1240bb921c146f099b9972 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:40.787679 5051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e546dde2b1240bb921c146f099b9972 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e546dde2b1240bb921c146f099b9972. Latest consensus state: current_term: 1 leader_uuid: "9e546dde2b1240bb921c146f099b9972" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e546dde2b1240bb921c146f099b9972" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39857 } } }
I20260926 09:27:40.787963 5051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e546dde2b1240bb921c146f099b9972 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:40.788009 5055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:27:40.788534 5055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:27:40.790134 5055 catalog_manager.cc:1347] Generated new cluster ID: 01e278f5a3f248dc804651f93e4ed1b5
I20260926 09:27:40.790179 5055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:27:40.802994 5055 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:27:40.803184 5055 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:27:40.815994 5055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e546dde2b1240bb921c146f099b9972: Generated new TSK 0
I20260926 09:27:40.816232 5055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:27:40.825863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39857
--builtin_ntp_servers=127.0.98.20:39321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 09:27:40.928970 5068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:40.929217 5068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:40.929257 5068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:40.930662 5068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:40.930724 5068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:40.930768 5068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:27:40.932253 5068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:40.932533 5068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:40.932850 5068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:40.938727 5074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:40.938733 5073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:40.938789 5076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:40.939122 5068 server_base.cc:1034] running on GCE node
I20260926 09:27:40.939344 5068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:40.939594 5068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:40.940757 5068 hybrid_clock.cc:648] HybridClock initialized: now 1790414860940742 us; error 31 us; skew 500 ppm
I20260926 09:27:40.942039 5068 webserver.cc:460] Webserver started at http://127.0.98.1:35535/ using document root <none> and password file <none>
I20260926 09:27:40.942306 5068 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:40.942376 5068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:40.942540 5068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:40.943631 5068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "7c54ce5045be46e7ad6960d9009e1727"
format_stamp: "Formatted at 2026-09-26 09:27:40 on dist-test-slave-dfzm"
server_key: "b908d00984ceb39d0f64e9fb885d1696"
server_key_iv: "49cbadeb7c47cd753aff9ab5c9abd471"
server_key_version: "encryptionkey@0"
I20260926 09:27:40.944021 5068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7c54ce5045be46e7ad6960d9009e1727"
format_stamp: "Formatted at 2026-09-26 09:27:40 on dist-test-slave-dfzm"
server_key: "b908d00984ceb39d0f64e9fb885d1696"
server_key_iv: "49cbadeb7c47cd753aff9ab5c9abd471"
server_key_version: "encryptionkey@0"
I20260926 09:27:40.945353 5068 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:27:40.946187 5082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:40.946359 5068 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:40.946467 5068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "7c54ce5045be46e7ad6960d9009e1727"
format_stamp: "Formatted at 2026-09-26 09:27:40 on dist-test-slave-dfzm"
server_key: "b908d00984ceb39d0f64e9fb885d1696"
server_key_iv: "49cbadeb7c47cd753aff9ab5c9abd471"
server_key_version: "encryptionkey@0"
I20260926 09:27:40.946544 5068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:40.950114 5068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:40.950416 5068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:40.950572 5068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:40.950788 5068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:40.951144 5068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:40.951192 5068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:40.951228 5068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:40.951256 5068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:40.957043 5068 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44085
I20260926 09:27:40.957118 5195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44085 every 8 connection(s)
I20260926 09:27:40.957453 5068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:27:40.960510 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:40.958237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34961 (local address 127.0.98.62:39857)
0926 09:27:40.958306 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:40.958309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:40.958592 (+ 283us) server_negotiation.cc:407] Connection header received
0926 09:27:40.959048 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:40.959049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:40.959057 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:40.959110 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:40.960306 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:40.960307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:40.960307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:27:40.960309 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:40.960320 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:27:40.960324 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:40.960365 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:40.960424 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 09:27:40.960452 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:27:40.960767 5198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:40.958243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.1:34961)
0926 09:27:40.958464 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:40.958498 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:40.958966 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:40.959163 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:40.959166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:40.959173 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:40.960173 (+ 1000us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:27:40.960174 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:40.960202 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:27:40.960231 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:40.960364 (+ 133us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:40.960367 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:40.960369 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:27:40.960422 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 09:27:40.960504 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":90,"threads_started":1}
I20260926 09:27:40.961284 5196 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39857
I20260926 09:27:40.961392 5196 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:40.961427 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 5068
I20260926 09:27:40.961555 5196 heartbeater.cc:511] Master 127.0.98.62:39857 requested a full tablet report, sending...
I20260926 09:27:40.961561 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:27:40.961781 392 external_mini_cluster.cc:1428] Setting key 9322fa23aee499b7254ec3d1a2773cbc
I20260926 09:27:40.962107 5008 ts_manager.cc:194] Registered new tserver with Master: 7c54ce5045be46e7ad6960d9009e1727 (127.0.98.1:44085)
I20260926 09:27:40.962878 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39857
--builtin_ntp_servers=127.0.98.20:39321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 09:27:41.044420 5199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:41.044637 5199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:41.044660 5199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:41.045842 5199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:41.045886 5199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:41.045912 5199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:27:41.047274 5199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:41.047463 5199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:41.047698 5199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:41.052775 5204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:41.052793 5205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:41.052781 5207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:41.053069 5199 server_base.cc:1034] running on GCE node
I20260926 09:27:41.053289 5199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:41.053535 5199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:41.054756 5199 hybrid_clock.cc:648] HybridClock initialized: now 1790414861054745 us; error 30 us; skew 500 ppm
I20260926 09:27:41.056154 5199 webserver.cc:460] Webserver started at http://127.0.98.2:35217/ using document root <none> and password file <none>
I20260926 09:27:41.056452 5199 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:41.056533 5199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:41.056684 5199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:41.057910 5199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "6b451e09e9df4bce8688f4cfa4fa162f"
format_stamp: "Formatted at 2026-09-26 09:27:41 on dist-test-slave-dfzm"
server_key: "5786c7eb79e27f2fa72ae000d07010fd"
server_key_iv: "de1b423235ae613aeff57ddb3441b9e6"
server_key_version: "encryptionkey@0"
I20260926 09:27:41.058284 5199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6b451e09e9df4bce8688f4cfa4fa162f"
format_stamp: "Formatted at 2026-09-26 09:27:41 on dist-test-slave-dfzm"
server_key: "5786c7eb79e27f2fa72ae000d07010fd"
server_key_iv: "de1b423235ae613aeff57ddb3441b9e6"
server_key_version: "encryptionkey@0"
I20260926 09:27:41.059669 5199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:27:41.060566 5213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:41.060756 5199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:41.060873 5199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "6b451e09e9df4bce8688f4cfa4fa162f"
format_stamp: "Formatted at 2026-09-26 09:27:41 on dist-test-slave-dfzm"
server_key: "5786c7eb79e27f2fa72ae000d07010fd"
server_key_iv: "de1b423235ae613aeff57ddb3441b9e6"
server_key_version: "encryptionkey@0"
I20260926 09:27:41.060966 5199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:41.064491 5199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:41.064776 5199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:41.064925 5199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:41.065147 5199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:41.065479 5199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:41.065536 5199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:41.065577 5199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:41.065626 5199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:41.071430 5199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32805
I20260926 09:27:41.071457 5326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32805 every 8 connection(s)
I20260926 09:27:41.071802 5199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:27:41.074921 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:41.072534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48985 (local address 127.0.98.62:39857)
0926 09:27:41.072600 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:41.072602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:41.072788 (+ 186us) server_negotiation.cc:407] Connection header received
0926 09:27:41.073269 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:41.073270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:41.073275 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:41.073340 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:41.074696 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:41.074697 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:41.074698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:27:41.074702 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:41.074717 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:27:41.074721 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:41.074777 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:41.074853 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 09:27:41.074866 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:27:41.075160 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:41.072488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.2:48985)
0926 09:27:41.072664 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:41.072708 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:41.073185 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:41.073419 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:41.073422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:41.073430 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:41.074547 (+ 1117us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:27:41.074547 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:41.074576 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:27:41.074605 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:41.074787 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:41.074791 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:41.074793 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:27:41.074851 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 09:27:41.074901 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260926 09:27:41.075668 5327 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39857
I20260926 09:27:41.075752 5327 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:41.075867 5327 heartbeater.cc:511] Master 127.0.98.62:39857 requested a full tablet report, sending...
I20260926 09:27:41.076222 5007 ts_manager.cc:194] Registered new tserver with Master: 6b451e09e9df4bce8688f4cfa4fa162f (127.0.98.2:32805)
I20260926 09:27:41.077428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 5199
I20260926 09:27:41.077526 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:27:41.077731 392 external_mini_cluster.cc:1428] Setting key 7dacedc153c855058d00ca2afa5a3ad7
I20260926 09:27:41.078636 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39857
--builtin_ntp_servers=127.0.98.20:39321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 09:27:41.160593 5330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:41.160815 5330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:41.160840 5330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:41.162036 5330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:41.162083 5330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:41.162111 5330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:27:41.163398 5330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:41.163689 5330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:41.163985 5330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:41.169139 5335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:41.169145 5338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:41.169189 5336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:41.169255 5330 server_base.cc:1034] running on GCE node
I20260926 09:27:41.169555 5330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:41.169831 5330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:41.170996 5330 hybrid_clock.cc:648] HybridClock initialized: now 1790414861170977 us; error 47 us; skew 500 ppm
I20260926 09:27:41.172309 5330 webserver.cc:460] Webserver started at http://127.0.98.3:39851/ using document root <none> and password file <none>
I20260926 09:27:41.172559 5330 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:41.172632 5330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:41.172770 5330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:41.173959 5330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "974c04c2755e429e959d4890c97a7094"
format_stamp: "Formatted at 2026-09-26 09:27:41 on dist-test-slave-dfzm"
server_key: "eaca641ce8de882aa11a32878ddb3ff3"
server_key_iv: "f3562a694ca16b8410dd5c637d6843ad"
server_key_version: "encryptionkey@0"
I20260926 09:27:41.174319 5330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "974c04c2755e429e959d4890c97a7094"
format_stamp: "Formatted at 2026-09-26 09:27:41 on dist-test-slave-dfzm"
server_key: "eaca641ce8de882aa11a32878ddb3ff3"
server_key_iv: "f3562a694ca16b8410dd5c637d6843ad"
server_key_version: "encryptionkey@0"
I20260926 09:27:41.175611 5330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:27:41.176296 5344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:41.176486 5330 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:41.176585 5330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "974c04c2755e429e959d4890c97a7094"
format_stamp: "Formatted at 2026-09-26 09:27:41 on dist-test-slave-dfzm"
server_key: "eaca641ce8de882aa11a32878ddb3ff3"
server_key_iv: "f3562a694ca16b8410dd5c637d6843ad"
server_key_version: "encryptionkey@0"
I20260926 09:27:41.176676 5330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:41.179997 5330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:41.180261 5330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:41.180402 5330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:41.180612 5330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:41.180912 5330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:41.180967 5330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:41.181010 5330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:41.181036 5330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:41.186707 5330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39209
I20260926 09:27:41.186743 5457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39209 every 8 connection(s)
I20260926 09:27:41.187158 5330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:27:41.190254 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:41.187917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44369 (local address 127.0.98.62:39857)
0926 09:27:41.187965 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:41.187967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:41.188323 (+ 356us) server_negotiation.cc:407] Connection header received
0926 09:27:41.188810 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:41.188810 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:41.188815 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:41.188870 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:41.190082 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:41.190083 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:41.190084 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:27:41.190086 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:41.190097 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:27:41.190101 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:41.190146 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:41.190190 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 09:27:41.190202 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:27:41.190487 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:41.187919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39857 (local address 127.0.98.3:44369)
0926 09:27:41.188140 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:41.188185 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:41.188719 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:41.188921 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:41.188924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:41.188932 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:41.189951 (+ 1019us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:27:41.189952 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:41.189979 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:27:41.190007 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:41.190143 (+ 136us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:41.190145 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:41.190148 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:27:41.190198 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 09:27:41.190243 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260926 09:27:41.190932 5458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39857
I20260926 09:27:41.191038 5458 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:41.191192 5458 heartbeater.cc:511] Master 127.0.98.62:39857 requested a full tablet report, sending...
I20260926 09:27:41.191581 5007 ts_manager.cc:194] Registered new tserver with Master: 974c04c2755e429e959d4890c97a7094 (127.0.98.3:39209)
I20260926 09:27:41.193463 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 5330
I20260926 09:27:41.193552 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:27:41.193751 392 external_mini_cluster.cc:1428] Setting key c0e04e36c2f4a2008b3018ada7f115d9
I20260926 09:27:41.194515 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:27:41.196767 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:41.195705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59072 (local address 127.0.98.62:39857)
0926 09:27:41.195765 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:41.195767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:41.195884 (+ 117us) server_negotiation.cc:407] Connection header received
0926 09:27:41.195923 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:41.195923 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:41.195927 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:41.195973 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:41.196571 (+ 598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:41.196571 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:41.196572 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:27:41.196574 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:41.196584 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:27:41.196587 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:41.196626 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:41.196658 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 09:27:41.196669 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 09:27:41.198287 5007 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:27:41.199759 5007 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:27:41.203989 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:41.202102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39209 (local address 127.0.0.1:34074)
0926 09:27:41.202293 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:41.202334 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:41.202394 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:41.203001 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:41.203007 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:41.203015 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:41.203547 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:27:41.203547 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:41.203563 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:27:41.203589 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:41.203813 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:41.203819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:41.203822 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:27:41.203892 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 09:27:41.203906 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":72,"threads_started":1}
I20260926 09:27:41.204149 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:41.202446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60710 (local address 127.0.98.1:44085)
0926 09:27:41.202619 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:41.202624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:41.202694 (+ 70us) server_negotiation.cc:407] Connection header received
0926 09:27:41.202764 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:41.202766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:41.202776 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:41.202875 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:41.203800 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:41.203808 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:41.203809 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:27:41.203813 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:41.203852 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:27:41.203881 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:41.203940 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:41.204040 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 09:27:41.204057 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260926 09:27:41.204309 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:41.202302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34074 (local address 127.0.98.3:39209)
0926 09:27:41.202838 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:41.202842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:41.202852 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:27:41.202874 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:41.202876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:41.202884 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:41.202954 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:41.203663 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:41.203670 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:41.203670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:27:41.203675 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:41.203719 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:27:41.203747 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:41.203821 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:41.204206 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 09:27:41.204224 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":87,"threads_started":1}
I20260926 09:27:41.204309 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:41.202202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44085 (local address 127.0.0.1:60710)
0926 09:27:41.202639 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:41.202648 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:41.202701 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:41.203222 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:41.203223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:41.203226 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:41.203696 (+ 470us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:27:41.203696 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:41.203701 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:27:41.203710 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:41.203937 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:41.203938 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:41.203939 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 09:27:41.204017 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 09:27:41.204031 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":102,"threads_started":1}
I20260926 09:27:41.204507 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:41.202102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32805 (local address 127.0.0.1:53706)
0926 09:27:41.202526 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:41.202540 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:41.202587 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:41.203471 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:41.203472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:41.203476 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:41.203882 (+ 406us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:27:41.203882 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:41.203889 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:27:41.203902 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:41.204306 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:41.204307 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:41.204308 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 09:27:41.204458 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 09:27:41.204469 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260926 09:27:41.205055 5392 tablet_service.cc:1467] Processing CreateTablet for tablet 7f0d601ae4e44b85b6cf76ebca44c285 (DEFAULT_TABLE table=test-table [id=f88006ab006a4e89b54ad713d38dd3b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:41.205054 5130 tablet_service.cc:1467] Processing CreateTablet for tablet 7f0d601ae4e44b85b6cf76ebca44c285 (DEFAULT_TABLE table=test-table [id=f88006ab006a4e89b54ad713d38dd3b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:41.205410 5130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f0d601ae4e44b85b6cf76ebca44c285. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:41.205410 5392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f0d601ae4e44b85b6cf76ebca44c285. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:41.208419 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:41.202314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53706 (local address 127.0.98.2:32805)
0926 09:27:41.203271 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:41.203275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:41.203285 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:27:41.203312 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:41.203313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:41.203323 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:41.203441 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:41.203991 (+ 550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:41.203998 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:41.203999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:27:41.204003 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:41.204050 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:27:41.204160 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:41.204306 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:41.208295 (+ 3989us) server_negotiation.cc:299] Negotiation successful
0926 09:27:41.208315 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":915,"thread_start_us":85,"threads_started":1}
I20260926 09:27:41.208825 5472 tablet_bootstrap.cc:492] T 7f0d601ae4e44b85b6cf76ebca44c285 P 7c54ce5045be46e7ad6960d9009e1727: Bootstrap starting.
I20260926 09:27:41.209061 5473 tablet_bootstrap.cc:492] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094: Bootstrap starting.
I20260926 09:27:41.209179 5261 tablet_service.cc:1467] Processing CreateTablet for tablet 7f0d601ae4e44b85b6cf76ebca44c285 (DEFAULT_TABLE table=test-table [id=f88006ab006a4e89b54ad713d38dd3b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:41.209544 5261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f0d601ae4e44b85b6cf76ebca44c285. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:41.209801 5473 tablet_bootstrap.cc:654] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:41.210129 5473 log.cc:824] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:41.210909 5473 tablet_bootstrap.cc:492] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094: No bootstrap required, opened a new log
I20260926 09:27:41.211058 5473 ts_tablet_manager.cc:1397] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:27:41.211063 5472 tablet_bootstrap.cc:654] T 7f0d601ae4e44b85b6cf76ebca44c285 P 7c54ce5045be46e7ad6960d9009e1727: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:41.211373 5472 log.cc:824] T 7f0d601ae4e44b85b6cf76ebca44c285 P 7c54ce5045be46e7ad6960d9009e1727: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:41.212173 5472 tablet_bootstrap.cc:492] T 7f0d601ae4e44b85b6cf76ebca44c285 P 7c54ce5045be46e7ad6960d9009e1727: No bootstrap required, opened a new log
I20260926 09:27:41.212248 5472 ts_tablet_manager.cc:1397] T 7f0d601ae4e44b85b6cf76ebca44c285 P 7c54ce5045be46e7ad6960d9009e1727: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 09:27:41.213027 5476 tablet_bootstrap.cc:492] T 7f0d601ae4e44b85b6cf76ebca44c285 P 6b451e09e9df4bce8688f4cfa4fa162f: Bootstrap starting.
I20260926 09:27:41.213523 5473 raft_consensus.cc:348] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974c04c2755e429e959d4890c97a7094" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39209 } } peers { permanent_uuid: "6b451e09e9df4bce8688f4cfa4fa162f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32805 } } peers { permanent_uuid: "7c54ce5045be46e7ad6960d9009e1727" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260926 09:27:41.213737 5473 raft_consensus.cc:374] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:41.213789 5473 raft_consensus.cc:729] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974c04c2755e429e959d4890c97a7094, State: Initialized, Role: FOLLOWER
I20260926 09:27:41.213826 5476 tablet_bootstrap.cc:654] T 7f0d601ae4e44b85b6cf76ebca44c285 P 6b451e09e9df4bce8688f4cfa4fa162f: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:41.213938 5473 consensus_queue.cc:260] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974c04c2755e429e959d4890c97a7094" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39209 } } peers { permanent_uuid: "6b451e09e9df4bce8688f4cfa4fa162f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32805 } } peers { permanent_uuid: "7c54ce5045be46e7ad6960d9009e1727" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260926 09:27:41.214210 5476 log.cc:824] T 7f0d601ae4e44b85b6cf76ebca44c285 P 6b451e09e9df4bce8688f4cfa4fa162f: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:41.214238 5473 ts_tablet_manager.cc:1428] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:27:41.214362 5458 heartbeater.cc:503] Master 127.0.98.62:39857 was elected leader, sending a full tablet report...
I20260926 09:27:41.215065 5476 tablet_bootstrap.cc:492] T 7f0d601ae4e44b85b6cf76ebca44c285 P 6b451e09e9df4bce8688f4cfa4fa162f: No bootstrap required, opened a new log
I20260926 09:27:41.215006 5472 raft_consensus.cc:348] T 7f0d601ae4e44b85b6cf76ebca44c285 P 7c54ce5045be46e7ad6960d9009e1727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974c04c2755e429e959d4890c97a7094" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39209 } } peers { permanent_uuid: "6b451e09e9df4bce8688f4cfa4fa162f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32805 } } peers { permanent_uuid: "7c54ce5045be46e7ad6960d9009e1727" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260926 09:27:41.215142 5476 ts_tablet_manager.cc:1397] T 7f0d601ae4e44b85b6cf76ebca44c285 P 6b451e09e9df4bce8688f4cfa4fa162f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:27:41.215147 5472 raft_consensus.cc:374] T 7f0d601ae4e44b85b6cf76ebca44c285 P 7c54ce5045be46e7ad6960d9009e1727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:41.215185 5472 raft_consensus.cc:729] T 7f0d601ae4e44b85b6cf76ebca44c285 P 7c54ce5045be46e7ad6960d9009e1727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c54ce5045be46e7ad6960d9009e1727, State: Initialized, Role: FOLLOWER
I20260926 09:27:41.215294 5472 consensus_queue.cc:260] T 7f0d601ae4e44b85b6cf76ebca44c285 P 7c54ce5045be46e7ad6960d9009e1727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974c04c2755e429e959d4890c97a7094" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39209 } } peers { permanent_uuid: "6b451e09e9df4bce8688f4cfa4fa162f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32805 } } peers { permanent_uuid: "7c54ce5045be46e7ad6960d9009e1727" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260926 09:27:41.215602 5472 ts_tablet_manager.cc:1428] T 7f0d601ae4e44b85b6cf76ebca44c285 P 7c54ce5045be46e7ad6960d9009e1727: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:27:41.215639 5196 heartbeater.cc:503] Master 127.0.98.62:39857 was elected leader, sending a full tablet report...
I20260926 09:27:41.217752 5476 raft_consensus.cc:348] T 7f0d601ae4e44b85b6cf76ebca44c285 P 6b451e09e9df4bce8688f4cfa4fa162f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974c04c2755e429e959d4890c97a7094" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39209 } } peers { permanent_uuid: "6b451e09e9df4bce8688f4cfa4fa162f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32805 } } peers { permanent_uuid: "7c54ce5045be46e7ad6960d9009e1727" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260926 09:27:41.217927 5476 raft_consensus.cc:374] T 7f0d601ae4e44b85b6cf76ebca44c285 P 6b451e09e9df4bce8688f4cfa4fa162f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:41.217996 5476 raft_consensus.cc:729] T 7f0d601ae4e44b85b6cf76ebca44c285 P 6b451e09e9df4bce8688f4cfa4fa162f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b451e09e9df4bce8688f4cfa4fa162f, State: Initialized, Role: FOLLOWER
I20260926 09:27:41.218117 5476 consensus_queue.cc:260] T 7f0d601ae4e44b85b6cf76ebca44c285 P 6b451e09e9df4bce8688f4cfa4fa162f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974c04c2755e429e959d4890c97a7094" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39209 } } peers { permanent_uuid: "6b451e09e9df4bce8688f4cfa4fa162f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32805 } } peers { permanent_uuid: "7c54ce5045be46e7ad6960d9009e1727" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260926 09:27:41.218384 5476 ts_tablet_manager.cc:1428] T 7f0d601ae4e44b85b6cf76ebca44c285 P 6b451e09e9df4bce8688f4cfa4fa162f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 09:27:41.218437 5327 heartbeater.cc:503] Master 127.0.98.62:39857 was elected leader, sending a full tablet report...
I20260926 09:27:41.224411 5477 raft_consensus.cc:482] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:27:41.224650 5477 raft_consensus.cc:504] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974c04c2755e429e959d4890c97a7094" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39209 } } peers { permanent_uuid: "6b451e09e9df4bce8688f4cfa4fa162f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32805 } } peers { permanent_uuid: "7c54ce5045be46e7ad6960d9009e1727" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260926 09:27:41.224993 5477 leader_election.cc:288] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b451e09e9df4bce8688f4cfa4fa162f (127.0.98.2:32805), 7c54ce5045be46e7ad6960d9009e1727 (127.0.98.1:44085)
I20260926 09:27:41.226471 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:41.225248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41285 (local address 127.0.98.2:32805)
0926 09:27:41.225387 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:41.225388 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:41.225395 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:27:41.225436 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:41.225437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:41.225443 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:41.225511 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:41.226227 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:41.226228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:41.226228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:27:41.226233 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:41.226249 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:27:41.226254 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:41.226316 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:41.226386 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 09:27:41.226400 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260926 09:27:41.226493 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:41.225173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32805 (local address 127.0.98.3:41285)
0926 09:27:41.225250 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:41.225263 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:41.225315 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:41.225524 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:41.225524 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:41.225527 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:41.226084 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:27:41.226084 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:41.226105 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:27:41.226120 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:41.226326 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:41.226326 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:41.226327 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 09:27:41.226378 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 09:27:41.226395 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 09:27:41.226471 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:41.225277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39571 (local address 127.0.98.1:44085)
0926 09:27:41.225337 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:41.225339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:41.225448 (+ 109us) server_negotiation.cc:407] Connection header received
0926 09:27:41.225516 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:41.225516 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:41.225522 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:41.225580 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:41.226227 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:41.226228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:41.226229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:27:41.226233 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:41.226249 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:27:41.226254 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:41.226316 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:41.226385 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 09:27:41.226400 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 09:27:41.226495 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:41.225173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44085 (local address 127.0.98.3:39571)
0926 09:27:41.225355 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:41.225366 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:41.225413 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:41.225599 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:41.225600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:41.225602 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:41.226136 (+ 534us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:27:41.226136 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:41.226141 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:27:41.226151 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:41.226326 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:41.226326 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:41.226327 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 09:27:41.226379 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 09:27:41.226396 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":22,"thread_start_us":101,"threads_started":1}
I20260926 09:27:41.226861 5281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0d601ae4e44b85b6cf76ebca44c285" candidate_uuid: "974c04c2755e429e959d4890c97a7094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b451e09e9df4bce8688f4cfa4fa162f" is_pre_election: true
I20260926 09:27:41.227017 5281 raft_consensus.cc:2454] T 7f0d601ae4e44b85b6cf76ebca44c285 P 6b451e09e9df4bce8688f4cfa4fa162f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974c04c2755e429e959d4890c97a7094 in term 0.
I20260926 09:27:41.227154 5150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0d601ae4e44b85b6cf76ebca44c285" candidate_uuid: "974c04c2755e429e959d4890c97a7094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c54ce5045be46e7ad6960d9009e1727" is_pre_election: true
I20260926 09:27:41.227221 5345 leader_election.cc:302] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b451e09e9df4bce8688f4cfa4fa162f, 974c04c2755e429e959d4890c97a7094; no voters:
I20260926 09:27:41.227288 5150 raft_consensus.cc:2454] T 7f0d601ae4e44b85b6cf76ebca44c285 P 7c54ce5045be46e7ad6960d9009e1727 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974c04c2755e429e959d4890c97a7094 in term 0.
I20260926 09:27:41.227350 5477 raft_consensus.cc:2781] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:27:41.227432 5477 raft_consensus.cc:482] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:27:41.227470 5477 raft_consensus.cc:3037] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:41.228184 5477 raft_consensus.cc:504] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974c04c2755e429e959d4890c97a7094" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39209 } } peers { permanent_uuid: "6b451e09e9df4bce8688f4cfa4fa162f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32805 } } peers { permanent_uuid: "7c54ce5045be46e7ad6960d9009e1727" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260926 09:27:41.228324 5477 leader_election.cc:288] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094 [CANDIDATE]: Term 1 election: Requested vote from peers 6b451e09e9df4bce8688f4cfa4fa162f (127.0.98.2:32805), 7c54ce5045be46e7ad6960d9009e1727 (127.0.98.1:44085)
I20260926 09:27:41.228478 5281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0d601ae4e44b85b6cf76ebca44c285" candidate_uuid: "974c04c2755e429e959d4890c97a7094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b451e09e9df4bce8688f4cfa4fa162f"
I20260926 09:27:41.228554 5281 raft_consensus.cc:3037] T 7f0d601ae4e44b85b6cf76ebca44c285 P 6b451e09e9df4bce8688f4cfa4fa162f [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:41.229230 5281 raft_consensus.cc:2454] T 7f0d601ae4e44b85b6cf76ebca44c285 P 6b451e09e9df4bce8688f4cfa4fa162f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974c04c2755e429e959d4890c97a7094 in term 1.
I20260926 09:27:41.229411 5345 leader_election.cc:302] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b451e09e9df4bce8688f4cfa4fa162f, 974c04c2755e429e959d4890c97a7094; no voters:
I20260926 09:27:41.229423 5150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0d601ae4e44b85b6cf76ebca44c285" candidate_uuid: "974c04c2755e429e959d4890c97a7094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c54ce5045be46e7ad6960d9009e1727"
I20260926 09:27:41.229502 5150 raft_consensus.cc:3037] T 7f0d601ae4e44b85b6cf76ebca44c285 P 7c54ce5045be46e7ad6960d9009e1727 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:41.229552 5477 raft_consensus.cc:2781] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:41.229717 5477 raft_consensus.cc:686] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094 [term 1 LEADER]: Becoming Leader. State: Replica: 974c04c2755e429e959d4890c97a7094, State: Running, Role: LEADER
I20260926 09:27:41.229831 5477 consensus_queue.cc:237] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974c04c2755e429e959d4890c97a7094" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39209 } } peers { permanent_uuid: "6b451e09e9df4bce8688f4cfa4fa162f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32805 } } peers { permanent_uuid: "7c54ce5045be46e7ad6960d9009e1727" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } }
I20260926 09:27:41.230428 5150 raft_consensus.cc:2454] T 7f0d601ae4e44b85b6cf76ebca44c285 P 7c54ce5045be46e7ad6960d9009e1727 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974c04c2755e429e959d4890c97a7094 in term 1.
I20260926 09:27:41.230696 5007 catalog_manager.cc:5495] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094 reported cstate change: term changed from 0 to 1, leader changed from <none> to 974c04c2755e429e959d4890c97a7094 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "974c04c2755e429e959d4890c97a7094" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974c04c2755e429e959d4890c97a7094" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b451e09e9df4bce8688f4cfa4fa162f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c54ce5045be46e7ad6960d9009e1727" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 } health_report { overall_health: UNKNOWN } } }
I20260926 09:27:41.241185 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:41.240157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34078 (local address 127.0.98.3:39209)
0926 09:27:41.240221 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:41.240223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:41.240234 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:27:41.240274 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:41.240275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:41.240288 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:41.240335 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:41.240958 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:41.240960 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:41.240960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:27:41.240965 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:41.240980 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:27:41.240984 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:41.241044 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:41.241096 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 09:27:41.241110 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:27:41.242944 5150 raft_consensus.cc:1261] T 7f0d601ae4e44b85b6cf76ebca44c285 P 7c54ce5045be46e7ad6960d9009e1727 [term 1 FOLLOWER]: Refusing update from remote peer 974c04c2755e429e959d4890c97a7094: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:41.242970 5281 raft_consensus.cc:1261] T 7f0d601ae4e44b85b6cf76ebca44c285 P 6b451e09e9df4bce8688f4cfa4fa162f [term 1 FOLLOWER]: Refusing update from remote peer 974c04c2755e429e959d4890c97a7094: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:41.243312 5477 consensus_queue.cc:1035] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c54ce5045be46e7ad6960d9009e1727" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:41.243436 5482 consensus_queue.cc:1035] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b451e09e9df4bce8688f4cfa4fa162f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:41.246074 5486 mvcc.cc:205] Tried to move back new op lower bound from 7333539271648542720 to 7333539271598120960. Current Snapshot: MvccSnapshot[applied={T|T < 7333539271648542720}]
I20260926 09:27:41.246546 5487 mvcc.cc:205] Tried to move back new op lower bound from 7333539271648542720 to 7333539271598120960. Current Snapshot: MvccSnapshot[applied={T|T < 7333539271648542720 or (T in {7333539271648542720})}]
I20260926 09:27:41.248250 5006 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:41.248350 5006 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:41.249133 5006 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e546dde2b1240bb921c146f099b9972: Sending DeleteTablet for 3 replicas of tablet 7f0d601ae4e44b85b6cf76ebca44c285
I20260926 09:27:41.249426 5392 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f0d601ae4e44b85b6cf76ebca44c285 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:41 UTC) from {username='slave'} at 127.0.0.1:34074
I20260926 09:27:41.249495 5261 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f0d601ae4e44b85b6cf76ebca44c285 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:41 UTC) from {username='slave'} at 127.0.0.1:53706
I20260926 09:27:41.249686 5495 tablet_replica.cc:330] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094: stopping tablet replica
I20260926 09:27:41.249737 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 5068
I20260926 09:27:41.249791 5495 raft_consensus.cc:2229] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:27:41.249786 5130 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f0d601ae4e44b85b6cf76ebca44c285 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:41 UTC) from {username='slave'} at 127.0.0.1:60710
I20260926 09:27:41.249913 5495 raft_consensus.cc:2258] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:41.251358 5495 ts_tablet_manager.cc:1905] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:41.253424 5495 ts_tablet_manager.cc:1918] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:27:41.253501 5495 log.cc:1196] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/7f0d601ae4e44b85b6cf76ebca44c285
I20260926 09:27:41.253799 5495 ts_tablet_manager.cc:1939] T 7f0d601ae4e44b85b6cf76ebca44c285 P 974c04c2755e429e959d4890c97a7094: Deleting consensus metadata
I20260926 09:27:41.254222 4993 catalog_manager.cc:4841] TS 974c04c2755e429e959d4890c97a7094 (127.0.98.3:39209): tablet 7f0d601ae4e44b85b6cf76ebca44c285 (table test-table [id=f88006ab006a4e89b54ad713d38dd3b1]) successfully deleted
W20260926 09:27:41.255584 4995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44085 (error 108)
I20260926 09:27:41.255805 5496 tablet_replica.cc:330] T 7f0d601ae4e44b85b6cf76ebca44c285 P 6b451e09e9df4bce8688f4cfa4fa162f: stopping tablet replica
I20260926 09:27:41.255932 5496 raft_consensus.cc:2229] T 7f0d601ae4e44b85b6cf76ebca44c285 P 6b451e09e9df4bce8688f4cfa4fa162f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:41.256044 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:41.255913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44085 (local address 127.0.0.1:60712)
0926 09:27:41.255968 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:41.256014 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 09:27:41.256054 5496 raft_consensus.cc:2258] T 7f0d601ae4e44b85b6cf76ebca44c285 P 6b451e09e9df4bce8688f4cfa4fa162f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 09:27:41.256158 4995 catalog_manager.cc:4584] TS 7c54ce5045be46e7ad6960d9009e1727 (127.0.98.1:44085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f0d601ae4e44b85b6cf76ebca44c285: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44085: connect: Connection refused (error 111)
I20260926 09:27:41.256248 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 5199
I20260926 09:27:41.261467 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:41.261339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32805 (local address 127.0.0.1:53710)
0926 09:27:41.261398 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:41.261430 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:32805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 09:27:41.261547 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 5330
I20260926 09:27:41.266475 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 4977
2026-09-26T09:27:41Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (694 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 09:27:41.286408 392 mini_oidc.cc:152] Starting JWKS server
I20260926 09:27:41.287532 392 webserver.cc:460] Webserver started at https://127.0.0.1:37943/ using document root <none> and password file <none>
I20260926 09:27:41.287665 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 09:27:41.287866 392 webserver.cc:460] Webserver started at http://127.0.0.1:41781/ using document root <none> and password file <none>
2026-09-26T09:27:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:27:41Z Disabled control of system clock
I20260926 09:27:41.296229 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45487
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45487
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37943/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414839069479-392-0/testchainca.cert with env {}
W20260926 09:27:41.378021 5503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:41.378234 5503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:41.378257 5503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:41.379426 5503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:27:41.379464 5503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:41.379477 5503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:41.379489 5503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:27:41.379501 5503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 09:27:41.379513 5503 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 09:27:41.379524 5503 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37943/jwks.json
I20260926 09:27:41.381072 5503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45487
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37943/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414839069479-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:41.381278 5503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:41.381448 5503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:41.386234 5511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:41.386273 5508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:41.386418 5503 server_base.cc:1034] running on GCE node
W20260926 09:27:41.386260 5509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:41.386754 5503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:41.387082 5503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:41.388252 5503 hybrid_clock.cc:648] HybridClock initialized: now 1790414861388224 us; error 60 us; skew 500 ppm
I20260926 09:27:41.389623 5503 webserver.cc:460] Webserver started at http://127.0.98.62:36663/ using document root <none> and password file <none>
I20260926 09:27:41.389881 5503 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:41.389955 5503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:41.390113 5503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:41.391225 5503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "af828873a7624b9391f2258f8768194d"
format_stamp: "Formatted at 2026-09-26 09:27:41 on dist-test-slave-dfzm"
server_key: "d57b8b83821849ac28cbcf575b8f978a"
server_key_iv: "d8a384121f498ee0152216c334f868ac"
server_key_version: "encryptionkey@0"
I20260926 09:27:41.391593 5503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "af828873a7624b9391f2258f8768194d"
format_stamp: "Formatted at 2026-09-26 09:27:41 on dist-test-slave-dfzm"
server_key: "d57b8b83821849ac28cbcf575b8f978a"
server_key_iv: "d8a384121f498ee0152216c334f868ac"
server_key_version: "encryptionkey@0"
I20260926 09:27:41.392867 5503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:27:41.393683 5517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:41.393862 5503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:41.393963 5503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "af828873a7624b9391f2258f8768194d"
format_stamp: "Formatted at 2026-09-26 09:27:41 on dist-test-slave-dfzm"
server_key: "d57b8b83821849ac28cbcf575b8f978a"
server_key_iv: "d8a384121f498ee0152216c334f868ac"
server_key_version: "encryptionkey@0"
I20260926 09:27:41.394066 5503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:41.412735 5503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:41.413110 5503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:41.413276 5503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:41.417315 5503 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45487
I20260926 09:27:41.417354 5569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45487 every 8 connection(s)
I20260926 09:27:41.417802 5503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:27:41.418804 5570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:41.420883 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 5503
I20260926 09:27:41.420972 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:27:41.421161 392 external_mini_cluster.cc:1428] Setting key ff51a1a9a832638602e1e57d71a5bda0
I20260926 09:27:41.421192 5570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af828873a7624b9391f2258f8768194d: Bootstrap starting.
I20260926 09:27:41.422041 5570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af828873a7624b9391f2258f8768194d: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:41.422364 5570 log.cc:824] T 00000000000000000000000000000000 P af828873a7624b9391f2258f8768194d: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:41.423216 5570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af828873a7624b9391f2258f8768194d: No bootstrap required, opened a new log
I20260926 09:27:41.425493 5570 raft_consensus.cc:348] T 00000000000000000000000000000000 P af828873a7624b9391f2258f8768194d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af828873a7624b9391f2258f8768194d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45487 } }
I20260926 09:27:41.425686 5570 raft_consensus.cc:374] T 00000000000000000000000000000000 P af828873a7624b9391f2258f8768194d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:41.425724 5570 raft_consensus.cc:729] T 00000000000000000000000000000000 P af828873a7624b9391f2258f8768194d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af828873a7624b9391f2258f8768194d, State: Initialized, Role: FOLLOWER
I20260926 09:27:41.425853 5570 consensus_queue.cc:260] T 00000000000000000000000000000000 P af828873a7624b9391f2258f8768194d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af828873a7624b9391f2258f8768194d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45487 } }
I20260926 09:27:41.425935 5570 raft_consensus.cc:388] T 00000000000000000000000000000000 P af828873a7624b9391f2258f8768194d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:27:41.425979 5570 raft_consensus.cc:482] T 00000000000000000000000000000000 P af828873a7624b9391f2258f8768194d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:27:41.426026 5570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af828873a7624b9391f2258f8768194d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:41.426829 5570 raft_consensus.cc:504] T 00000000000000000000000000000000 P af828873a7624b9391f2258f8768194d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af828873a7624b9391f2258f8768194d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45487 } }
I20260926 09:27:41.426945 5570 leader_election.cc:302] T 00000000000000000000000000000000 P af828873a7624b9391f2258f8768194d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af828873a7624b9391f2258f8768194d; no voters:
I20260926 09:27:41.427146 5570 leader_election.cc:288] T 00000000000000000000000000000000 P af828873a7624b9391f2258f8768194d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:27:41.427201 5575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af828873a7624b9391f2258f8768194d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:41.427369 5575 raft_consensus.cc:686] T 00000000000000000000000000000000 P af828873a7624b9391f2258f8768194d [term 1 LEADER]: Becoming Leader. State: Replica: af828873a7624b9391f2258f8768194d, State: Running, Role: LEADER
I20260926 09:27:41.427434 5570 sys_catalog.cc:564] T 00000000000000000000000000000000 P af828873a7624b9391f2258f8768194d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:27:41.427472 5575 consensus_queue.cc:237] T 00000000000000000000000000000000 P af828873a7624b9391f2258f8768194d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af828873a7624b9391f2258f8768194d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45487 } }
I20260926 09:27:41.427804 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:41.421723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60726 (local address 127.0.98.62:45487)
0926 09:27:41.421947 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:41.421956 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:41.421992 (+ 36us) server_negotiation.cc:407] Connection header received
0926 09:27:41.422593 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:41.422603 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:41.422681 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:41.422800 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:41.423035 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:41.424406 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:41.425219 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:41.425888 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:41.426014 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:41.426785 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:41.426790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:41.426790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:27:41.427007 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:41.427053 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:27:41.427069 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:41.427140 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:41.427326 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 09:27:41.427411 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260926 09:27:41.427834 5577 sys_catalog.cc:455] T 00000000000000000000000000000000 P af828873a7624b9391f2258f8768194d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af828873a7624b9391f2258f8768194d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af828873a7624b9391f2258f8768194d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45487 } } }
I20260926 09:27:41.427920 5577 sys_catalog.cc:458] T 00000000000000000000000000000000 P af828873a7624b9391f2258f8768194d [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:41.427817 5575 sys_catalog.cc:455] T 00000000000000000000000000000000 P af828873a7624b9391f2258f8768194d [sys.catalog]: SysCatalogTable state changed. Reason: New leader af828873a7624b9391f2258f8768194d. Latest consensus state: current_term: 1 leader_uuid: "af828873a7624b9391f2258f8768194d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af828873a7624b9391f2258f8768194d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45487 } } }
I20260926 09:27:41.428372 5575 sys_catalog.cc:458] T 00000000000000000000000000000000 P af828873a7624b9391f2258f8768194d [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:41.428532 5584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:27:41.428926 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:27:41.429019 5584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:27:41.434594 5584 catalog_manager.cc:1347] Generated new cluster ID: a2e9cebd86a6495eb3ca5d160fcd0871
I20260926 09:27:41.434659 5584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:27:41.450984 5584 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:27:41.451536 5584 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:27:41.462841 5584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af828873a7624b9391f2258f8768194d: Generated new TSK 0
I20260926 09:27:41.463073 5584 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:27:41.498400 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:41.475782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60730 (local address 127.0.98.62:45487)
0926 09:27:41.475848 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:41.475850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:41.475995 (+ 145us) server_negotiation.cc:407] Connection header received
0926 09:27:41.476102 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:41.476103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:41.476112 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:41.476182 (+ 70us) server_negotiation.cc:226] Negotiated authn=JWT
0926 09:27:41.476342 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:41.476833 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:41.478057 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:41.478252 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:41.478310 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:41.478547 (+ 237us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 09:27:41.498169 (+ 19622us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37943/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 09:27:41.498321 (+ 152us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60730: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37943/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 09:27:41.498602 5573 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60730: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37943/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 09:27:41.498801 5599 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:41.475723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45487 (local address 127.0.0.1:60730)
0926 09:27:41.475918 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:41.475930 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:41.476006 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:41.476206 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:41.476207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:41.476210 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0926 09:27:41.476280 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:41.476282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:41.476925 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:41.476926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:41.477962 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:41.477967 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:41.478331 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:41.478335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:41.478404 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:41.478426 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 09:27:41.498335 (+ 19909us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37943/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 09:27:41.498453 (+ 118us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45487: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37943/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":101,"threads_started":1}
E20260926 09:27:41.498898 5600 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 09:27:41.498958 5600 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:45487: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37943/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 09:27:41.499408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 5503
2026-09-26T09:27:41Z chronyd exiting
I20260926 09:27:41.535658 392 test_util.cc:175] -----------------------------------------------
I20260926 09:27:41.535745 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414839069479-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (250 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 09:27:41.536509 392 mini_oidc.cc:152] Starting JWKS server
I20260926 09:27:41.537220 392 webserver.cc:460] Webserver started at https://127.0.0.1:43721/ using document root <none> and password file <none>
I20260926 09:27:41.537261 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 09:27:41.537410 392 webserver.cc:460] Webserver started at http://127.0.0.1:40321/ using document root <none> and password file <none>
2026-09-26T09:27:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:27:41Z Disabled control of system clock
I20260926 09:27:41.544971 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36993
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36993
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43721/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414839069479-392-0/testchainca.cert with env {}
W20260926 09:27:41.624934 5606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:41.625195 5606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:41.625233 5606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:41.626469 5606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:27:41.626538 5606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:41.626559 5606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:41.626579 5606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:27:41.626611 5606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 09:27:41.626636 5606 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 09:27:41.626664 5606 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43721/jwks.json
I20260926 09:27:41.628547 5606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36993
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43721/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414839069479-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:41.628866 5606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:41.629103 5606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:41.634243 5614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:41.634261 5612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:41.634274 5606 server_base.cc:1034] running on GCE node
W20260926 09:27:41.634243 5611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:41.634788 5606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:41.635145 5606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:41.636322 5606 hybrid_clock.cc:648] HybridClock initialized: now 1790414861636316 us; error 36 us; skew 500 ppm
I20260926 09:27:41.637718 5606 webserver.cc:460] Webserver started at http://127.0.98.62:34857/ using document root <none> and password file <none>
I20260926 09:27:41.637969 5606 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:41.638046 5606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:41.638182 5606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:41.639281 5606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "730da573ed5d426ebdaa14b68044768b"
format_stamp: "Formatted at 2026-09-26 09:27:41 on dist-test-slave-dfzm"
server_key: "63352c42827e29dbcbd9bbf37aec3e76"
server_key_iv: "34809313d4d3e19f24a8b0843829b7d3"
server_key_version: "encryptionkey@0"
I20260926 09:27:41.639659 5606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "730da573ed5d426ebdaa14b68044768b"
format_stamp: "Formatted at 2026-09-26 09:27:41 on dist-test-slave-dfzm"
server_key: "63352c42827e29dbcbd9bbf37aec3e76"
server_key_iv: "34809313d4d3e19f24a8b0843829b7d3"
server_key_version: "encryptionkey@0"
I20260926 09:27:41.640939 5606 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:27:41.641775 5620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:41.641963 5606 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:41.642076 5606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "730da573ed5d426ebdaa14b68044768b"
format_stamp: "Formatted at 2026-09-26 09:27:41 on dist-test-slave-dfzm"
server_key: "63352c42827e29dbcbd9bbf37aec3e76"
server_key_iv: "34809313d4d3e19f24a8b0843829b7d3"
server_key_version: "encryptionkey@0"
I20260926 09:27:41.642172 5606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:41.668421 5606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:41.668797 5606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:41.668989 5606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:41.673442 5606 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36993
I20260926 09:27:41.673482 5672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36993 every 8 connection(s)
I20260926 09:27:41.673900 5606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:27:41.674925 5673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:41.677125 5673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 730da573ed5d426ebdaa14b68044768b: Bootstrap starting.
I20260926 09:27:41.677732 5673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 730da573ed5d426ebdaa14b68044768b: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:41.677965 5673 log.cc:824] T 00000000000000000000000000000000 P 730da573ed5d426ebdaa14b68044768b: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:41.678637 5673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 730da573ed5d426ebdaa14b68044768b: No bootstrap required, opened a new log
I20260926 09:27:41.679114 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 5606
I20260926 09:27:41.679241 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:27:41.679414 392 external_mini_cluster.cc:1428] Setting key 491f0668a85403f1e1f391d950c6145c
I20260926 09:27:41.680801 5673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 730da573ed5d426ebdaa14b68044768b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730da573ed5d426ebdaa14b68044768b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } }
I20260926 09:27:41.680974 5673 raft_consensus.cc:374] T 00000000000000000000000000000000 P 730da573ed5d426ebdaa14b68044768b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:41.681011 5673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 730da573ed5d426ebdaa14b68044768b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730da573ed5d426ebdaa14b68044768b, State: Initialized, Role: FOLLOWER
I20260926 09:27:41.681156 5673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 730da573ed5d426ebdaa14b68044768b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730da573ed5d426ebdaa14b68044768b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } }
I20260926 09:27:41.681246 5673 raft_consensus.cc:388] T 00000000000000000000000000000000 P 730da573ed5d426ebdaa14b68044768b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:27:41.681277 5673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 730da573ed5d426ebdaa14b68044768b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:27:41.681315 5673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 730da573ed5d426ebdaa14b68044768b [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:41.682088 5673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 730da573ed5d426ebdaa14b68044768b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730da573ed5d426ebdaa14b68044768b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } }
I20260926 09:27:41.682195 5673 leader_election.cc:302] T 00000000000000000000000000000000 P 730da573ed5d426ebdaa14b68044768b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 730da573ed5d426ebdaa14b68044768b; no voters:
I20260926 09:27:41.682343 5673 leader_election.cc:288] T 00000000000000000000000000000000 P 730da573ed5d426ebdaa14b68044768b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:27:41.682404 5678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 730da573ed5d426ebdaa14b68044768b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:41.682570 5678 raft_consensus.cc:686] T 00000000000000000000000000000000 P 730da573ed5d426ebdaa14b68044768b [term 1 LEADER]: Becoming Leader. State: Replica: 730da573ed5d426ebdaa14b68044768b, State: Running, Role: LEADER
I20260926 09:27:41.682578 5673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 730da573ed5d426ebdaa14b68044768b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:27:41.682687 5678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 730da573ed5d426ebdaa14b68044768b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730da573ed5d426ebdaa14b68044768b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } }
I20260926 09:27:41.684043 5680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 730da573ed5d426ebdaa14b68044768b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 730da573ed5d426ebdaa14b68044768b. Latest consensus state: current_term: 1 leader_uuid: "730da573ed5d426ebdaa14b68044768b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730da573ed5d426ebdaa14b68044768b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } } }
I20260926 09:27:41.684158 5680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 730da573ed5d426ebdaa14b68044768b [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:41.684537 5679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 730da573ed5d426ebdaa14b68044768b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "730da573ed5d426ebdaa14b68044768b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730da573ed5d426ebdaa14b68044768b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } } }
I20260926 09:27:41.684607 5679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 730da573ed5d426ebdaa14b68044768b [sys.catalog]: This master's current role is: LEADER
W20260926 09:27:41.684732 5693 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 730da573ed5d426ebdaa14b68044768b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:27:41.684778 5693 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:27:41.684827 5694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:27:41.684960 5694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:27:41.686522 5694 catalog_manager.cc:1347] Generated new cluster ID: 9143a6d6563e407e9b028bbcfe806269
I20260926 09:27:41.686587 5694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:27:41.686674 5677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:41.679907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53856 (local address 127.0.98.62:36993)
0926 09:27:41.680141 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:41.680150 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:41.680183 (+ 33us) server_negotiation.cc:407] Connection header received
0926 09:27:41.680765 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:41.680774 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:41.680853 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:41.681018 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:41.681257 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:41.683053 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:41.684170 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:41.684836 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:41.684951 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:41.685899 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:41.685903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:41.685904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:27:41.686121 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:41.686155 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:27:41.686167 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:41.686245 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:41.686327 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 09:27:41.686409 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":90,"threads_started":1}
I20260926 09:27:41.687485 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:27:41.722117 5694 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:27:41.722687 5694 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:27:41.727250 5694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 730da573ed5d426ebdaa14b68044768b: Generated new TSK 0
I20260926 09:27:41.727469 5694 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 09:27:41.754083 5703 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 09:27:41.754174 5703 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 09:27:41.754565 5677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:41.732391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53860 (local address 127.0.98.62:36993)
0926 09:27:41.732460 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:41.732463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:41.732588 (+ 125us) server_negotiation.cc:407] Connection header received
0926 09:27:41.732671 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:41.732672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:41.732684 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:41.732765 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0926 09:27:41.732928 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:41.733426 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:41.734277 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:41.734473 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:41.734565 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:41.734753 (+ 188us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 09:27:41.754316 (+ 19563us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43721/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 09:27:41.754484 (+ 168us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53860: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43721/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 09:27:41.754630 5677 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53860: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43721/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 09:27:41.754815 5702 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:41.732328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36993 (local address 127.0.0.1:53860)
0926 09:27:41.732510 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:41.732522 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:41.732606 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:41.732778 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:41.732779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:41.732781 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 09:27:41.732838 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:41.732840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:41.733516 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:41.733517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:41.734181 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:41.734184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:41.734568 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:41.734571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:41.734630 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:41.734635 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 09:27:41.754483 (+ 19848us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43721/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 09:27:41.754609 (+ 126us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36993: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43721/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":100,"threads_started":1}
I20260926 09:27:41.755116 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 5606
2026-09-26T09:27:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (255 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 09:27:41.792246 392 mini_oidc.cc:152] Starting JWKS server
I20260926 09:27:41.792987 392 webserver.cc:460] Webserver started at https://127.0.0.1:33215/ using document root <none> and password file <none>
I20260926 09:27:41.793058 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 09:27:41.793247 392 webserver.cc:460] Webserver started at http://127.0.0.1:34991/ using document root <none> and password file <none>
2026-09-26T09:27:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:27:41Z Disabled control of system clock
I20260926 09:27:41.801008 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42929
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42929
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33215/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 09:27:41.881212 5709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:41.881448 5709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:41.881471 5709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:41.882810 5709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:27:41.882863 5709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:41.882876 5709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:41.882889 5709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:27:41.882901 5709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 09:27:41.882920 5709 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 09:27:41.882931 5709 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33215/jwks.json
I20260926 09:27:41.884428 5709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42929
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33215/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:41.884702 5709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:41.885035 5709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:41.889914 5714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:41.889946 5717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:41.889918 5715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:41.890161 5709 server_base.cc:1034] running on GCE node
I20260926 09:27:41.890403 5709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:41.890740 5709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:41.891922 5709 hybrid_clock.cc:648] HybridClock initialized: now 1790414861891897 us; error 43 us; skew 500 ppm
I20260926 09:27:41.893311 5709 webserver.cc:460] Webserver started at http://127.0.98.62:38923/ using document root <none> and password file <none>
I20260926 09:27:41.893627 5709 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:41.893718 5709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:41.893880 5709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:41.894927 5709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "64aa563623c8445fa4e434378eefab1e"
format_stamp: "Formatted at 2026-09-26 09:27:41 on dist-test-slave-dfzm"
server_key: "4e6c8c3e833164bc3f6016f6e632347f"
server_key_iv: "4adb479724790394b2cfb8304ac847bb"
server_key_version: "encryptionkey@0"
I20260926 09:27:41.895237 5709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "64aa563623c8445fa4e434378eefab1e"
format_stamp: "Formatted at 2026-09-26 09:27:41 on dist-test-slave-dfzm"
server_key: "4e6c8c3e833164bc3f6016f6e632347f"
server_key_iv: "4adb479724790394b2cfb8304ac847bb"
server_key_version: "encryptionkey@0"
I20260926 09:27:41.896485 5709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:27:41.897176 5723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:41.897351 5709 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:41.897420 5709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "64aa563623c8445fa4e434378eefab1e"
format_stamp: "Formatted at 2026-09-26 09:27:41 on dist-test-slave-dfzm"
server_key: "4e6c8c3e833164bc3f6016f6e632347f"
server_key_iv: "4adb479724790394b2cfb8304ac847bb"
server_key_version: "encryptionkey@0"
I20260926 09:27:41.897475 5709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:41.906586 5709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:41.906870 5709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:41.906975 5709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:41.911281 5709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42929
I20260926 09:27:41.911304 5775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42929 every 8 connection(s)
I20260926 09:27:41.911676 5709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:27:41.912608 5776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:41.915548 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 5709
I20260926 09:27:41.915654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:27:41.915846 392 external_mini_cluster.cc:1428] Setting key 6446a614a91b4e96154a3cdccc181e55
I20260926 09:27:41.916085 5776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64aa563623c8445fa4e434378eefab1e: Bootstrap starting.
I20260926 09:27:41.916893 5776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64aa563623c8445fa4e434378eefab1e: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:41.917225 5776 log.cc:824] T 00000000000000000000000000000000 P 64aa563623c8445fa4e434378eefab1e: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:41.918015 5776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64aa563623c8445fa4e434378eefab1e: No bootstrap required, opened a new log
I20260926 09:27:41.919814 5776 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64aa563623c8445fa4e434378eefab1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64aa563623c8445fa4e434378eefab1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42929 } }
I20260926 09:27:41.919986 5776 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64aa563623c8445fa4e434378eefab1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:41.920034 5776 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64aa563623c8445fa4e434378eefab1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64aa563623c8445fa4e434378eefab1e, State: Initialized, Role: FOLLOWER
I20260926 09:27:41.920179 5776 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64aa563623c8445fa4e434378eefab1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64aa563623c8445fa4e434378eefab1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42929 } }
I20260926 09:27:41.920280 5776 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64aa563623c8445fa4e434378eefab1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:27:41.920325 5776 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64aa563623c8445fa4e434378eefab1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:27:41.920373 5776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64aa563623c8445fa4e434378eefab1e [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:41.920951 5776 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64aa563623c8445fa4e434378eefab1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64aa563623c8445fa4e434378eefab1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42929 } }
I20260926 09:27:41.921082 5776 leader_election.cc:302] T 00000000000000000000000000000000 P 64aa563623c8445fa4e434378eefab1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64aa563623c8445fa4e434378eefab1e; no voters:
I20260926 09:27:41.921271 5776 leader_election.cc:288] T 00000000000000000000000000000000 P 64aa563623c8445fa4e434378eefab1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:27:41.921299 5781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64aa563623c8445fa4e434378eefab1e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:41.921409 5781 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64aa563623c8445fa4e434378eefab1e [term 1 LEADER]: Becoming Leader. State: Replica: 64aa563623c8445fa4e434378eefab1e, State: Running, Role: LEADER
I20260926 09:27:41.921495 5781 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64aa563623c8445fa4e434378eefab1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64aa563623c8445fa4e434378eefab1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42929 } }
I20260926 09:27:41.921571 5776 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64aa563623c8445fa4e434378eefab1e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:27:41.921871 5783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64aa563623c8445fa4e434378eefab1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64aa563623c8445fa4e434378eefab1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64aa563623c8445fa4e434378eefab1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42929 } } }
I20260926 09:27:41.922061 5783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64aa563623c8445fa4e434378eefab1e [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:41.922003 5782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64aa563623c8445fa4e434378eefab1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64aa563623c8445fa4e434378eefab1e. Latest consensus state: current_term: 1 leader_uuid: "64aa563623c8445fa4e434378eefab1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64aa563623c8445fa4e434378eefab1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42929 } } }
I20260926 09:27:41.922251 5779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:41.916362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.0.98.62:42929)
0926 09:27:41.916596 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:41.916606 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:41.917087 (+ 481us) server_negotiation.cc:407] Connection header received
0926 09:27:41.917722 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:41.917730 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:41.917794 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:41.917951 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:41.918151 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:41.919060 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:41.919863 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:41.920427 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:41.920544 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:41.921280 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:41.921284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:41.921284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:27:41.921507 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:41.921554 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:27:41.921570 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:41.921887 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:41.921920 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 09:27:41.922012 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":92,"threads_started":1}
I20260926 09:27:41.922339 5782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64aa563623c8445fa4e434378eefab1e [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:41.922650 5788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:27:41.923115 5788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:27:41.923233 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:27:41.924904 5788 catalog_manager.cc:1347] Generated new cluster ID: 9cc96af7f1b04a12b9374c101bee2e82
I20260926 09:27:41.924958 5788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:27:41.935065 5788 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:27:41.935775 5788 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:27:41.941506 5788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64aa563623c8445fa4e434378eefab1e: Generated new TSK 0
I20260926 09:27:41.941758 5788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:27:41.959489 5779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:41.955704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38174 (local address 127.0.98.62:42929)
0926 09:27:41.955777 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:41.955780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:41.955921 (+ 141us) server_negotiation.cc:407] Connection header received
0926 09:27:41.955964 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:41.955965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:41.955978 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:41.956047 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0926 09:27:41.956207 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:41.956729 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:41.957887 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:41.958074 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:41.958128 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:41.958324 (+ 196us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 09:27:41.959271 (+ 947us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33215/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 09:27:41.959404 (+ 133us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38174: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33215/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 09:27:41.959560 5779 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38174: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33215/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 09:27:41.959690 5805 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:41.955637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42929 (local address 127.0.0.1:38174)
0926 09:27:41.955810 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:41.955822 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:41.955895 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:41.956070 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:41.956071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:41.956074 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0926 09:27:41.956137 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:41.956139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:41.956808 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:41.956809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:41.957802 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:41.957806 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:41.958149 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:41.958157 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:41.958217 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:41.958223 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 09:27:41.959373 (+ 1150us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33215/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 09:27:41.959477 (+ 104us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42929: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33215/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":120,"threads_started":1}
I20260926 09:27:41.960034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 5709
E20260926 09:27:41.970721 5806 webserver.cc:533] Webserver: sslize failed:
E20260926 09:27:41.970808 5806 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T09:27:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (205 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T09:27:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:27:42Z Disabled control of system clock
I20260926 09:27:42.004690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37389
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37389
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414839069479-392-0/testchainca.cert with env {}
W20260926 09:27:42.085059 5810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:42.085296 5810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:42.085318 5810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:42.086458 5810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:27:42.086498 5810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:42.086511 5810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:42.086524 5810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:27:42.086535 5810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 09:27:42.086548 5810 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 09:27:42.086560 5810 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 09:27:42.087998 5810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37389
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414839069479-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:42.088210 5810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:42.088378 5810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:42.093415 5816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:42.093405 5818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:42.093405 5815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:42.093659 5810 server_base.cc:1034] running on GCE node
I20260926 09:27:42.093897 5810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:42.094254 5810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:42.095470 5810 hybrid_clock.cc:648] HybridClock initialized: now 1790414862095451 us; error 48 us; skew 500 ppm
I20260926 09:27:42.096854 5810 webserver.cc:460] Webserver started at http://127.0.98.62:39725/ using document root <none> and password file <none>
I20260926 09:27:42.097118 5810 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:42.097193 5810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:42.097355 5810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:42.098591 5810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "637d7242d3bc498caf779a5dc3333284"
format_stamp: "Formatted at 2026-09-26 09:27:42 on dist-test-slave-dfzm"
server_key: "c094e62ff5eb7686668d6f79efa6868c"
server_key_iv: "f0277d61ed4a0e6197eb9e44417006a7"
server_key_version: "encryptionkey@0"
I20260926 09:27:42.098975 5810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "637d7242d3bc498caf779a5dc3333284"
format_stamp: "Formatted at 2026-09-26 09:27:42 on dist-test-slave-dfzm"
server_key: "c094e62ff5eb7686668d6f79efa6868c"
server_key_iv: "f0277d61ed4a0e6197eb9e44417006a7"
server_key_version: "encryptionkey@0"
I20260926 09:27:42.100332 5810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:27:42.101059 5824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:42.101231 5810 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:42.101333 5810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "637d7242d3bc498caf779a5dc3333284"
format_stamp: "Formatted at 2026-09-26 09:27:42 on dist-test-slave-dfzm"
server_key: "c094e62ff5eb7686668d6f79efa6868c"
server_key_iv: "f0277d61ed4a0e6197eb9e44417006a7"
server_key_version: "encryptionkey@0"
I20260926 09:27:42.101423 5810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:42.110154 5810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:42.110478 5810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:42.110632 5810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:42.114557 5810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37389
I20260926 09:27:42.114603 5876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37389 every 8 connection(s)
I20260926 09:27:42.114985 5810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:27:42.115960 5877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:42.118340 5877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 637d7242d3bc498caf779a5dc3333284: Bootstrap starting.
I20260926 09:27:42.118917 5877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 637d7242d3bc498caf779a5dc3333284: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:42.119187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 5810
I20260926 09:27:42.119212 5877 log.cc:824] T 00000000000000000000000000000000 P 637d7242d3bc498caf779a5dc3333284: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:42.119326 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:27:42.119593 392 external_mini_cluster.cc:1428] Setting key eabecc05dfc15cac4ca74553c58caca6
I20260926 09:27:42.119925 5877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 637d7242d3bc498caf779a5dc3333284: No bootstrap required, opened a new log
I20260926 09:27:42.121747 5877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 637d7242d3bc498caf779a5dc3333284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "637d7242d3bc498caf779a5dc3333284" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37389 } }
I20260926 09:27:42.121908 5877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 637d7242d3bc498caf779a5dc3333284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:42.121951 5877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 637d7242d3bc498caf779a5dc3333284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 637d7242d3bc498caf779a5dc3333284, State: Initialized, Role: FOLLOWER
I20260926 09:27:42.122119 5877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 637d7242d3bc498caf779a5dc3333284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "637d7242d3bc498caf779a5dc3333284" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37389 } }
I20260926 09:27:42.122211 5877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 637d7242d3bc498caf779a5dc3333284 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:27:42.122256 5877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 637d7242d3bc498caf779a5dc3333284 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:27:42.122308 5877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 637d7242d3bc498caf779a5dc3333284 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:42.122846 5877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 637d7242d3bc498caf779a5dc3333284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "637d7242d3bc498caf779a5dc3333284" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37389 } }
I20260926 09:27:42.122962 5877 leader_election.cc:302] T 00000000000000000000000000000000 P 637d7242d3bc498caf779a5dc3333284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 637d7242d3bc498caf779a5dc3333284; no voters:
I20260926 09:27:42.123159 5877 leader_election.cc:288] T 00000000000000000000000000000000 P 637d7242d3bc498caf779a5dc3333284 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:27:42.123181 5882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 637d7242d3bc498caf779a5dc3333284 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:42.123293 5882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 637d7242d3bc498caf779a5dc3333284 [term 1 LEADER]: Becoming Leader. State: Replica: 637d7242d3bc498caf779a5dc3333284, State: Running, Role: LEADER
I20260926 09:27:42.123373 5882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 637d7242d3bc498caf779a5dc3333284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "637d7242d3bc498caf779a5dc3333284" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37389 } }
I20260926 09:27:42.123430 5877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 637d7242d3bc498caf779a5dc3333284 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:27:42.123701 5884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 637d7242d3bc498caf779a5dc3333284 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "637d7242d3bc498caf779a5dc3333284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "637d7242d3bc498caf779a5dc3333284" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37389 } } }
I20260926 09:27:42.123849 5884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 637d7242d3bc498caf779a5dc3333284 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:42.124128 5889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:27:42.124629 5889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:27:42.123705 5883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 637d7242d3bc498caf779a5dc3333284 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 637d7242d3bc498caf779a5dc3333284. Latest consensus state: current_term: 1 leader_uuid: "637d7242d3bc498caf779a5dc3333284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "637d7242d3bc498caf779a5dc3333284" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37389 } } }
I20260926 09:27:42.125404 5883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 637d7242d3bc498caf779a5dc3333284 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:42.126835 5889 catalog_manager.cc:1347] Generated new cluster ID: eb057989c0354a88af1ee0f6c6424cc7
I20260926 09:27:42.126894 5889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:27:42.127332 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:42.120159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60184 (local address 127.0.98.62:37389)
0926 09:27:42.120408 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:42.120416 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:42.120442 (+ 26us) server_negotiation.cc:407] Connection header received
0926 09:27:42.121744 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:42.121753 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:42.121820 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:42.121976 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:42.122198 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:42.123715 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:42.125017 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:42.125664 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:42.125797 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:42.126567 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:42.126571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:42.126571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:27:42.126803 (+ 232us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:42.126845 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:27:42.126862 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:42.126932 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:42.127042 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 09:27:42.127127 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":85,"threads_started":1}
I20260926 09:27:42.128273 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:27:42.159521 5889 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:27:42.160094 5889 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:27:42.166386 5889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 637d7242d3bc498caf779a5dc3333284: Generated new TSK 0
I20260926 09:27:42.166617 5889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:27:42.647394 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:42.183150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60188 (local address 127.0.98.62:37389)
0926 09:27:42.183215 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:42.183218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:42.183302 (+ 84us) server_negotiation.cc:407] Connection header received
0926 09:27:42.183365 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:42.183366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:42.183379 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:42.183461 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0926 09:27:42.183655 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:42.184362 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:42.185235 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:42.185575 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:42.185687 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:42.185933 (+ 246us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 09:27:42.647171 (+461238us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 09:27:42.647333 (+ 162us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60188: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 09:27:42.647501 5881 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60188: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 09:27:42.647580 5906 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:42.183061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37389 (local address 127.0.0.1:60188)
0926 09:27:42.183225 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:42.183235 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:42.183307 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:42.183508 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:42.183509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:42.183511 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 09:27:42.183570 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:42.183572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:42.184513 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:42.184514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:42.185131 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:42.185133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:42.185726 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:42.185732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:42.185788 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:42.185793 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 09:27:42.647335 (+461542us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 09:27:42.647455 (+ 120us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37389: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":109,"threads_started":1}
I20260926 09:27:42.647926 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 5810
2026-09-26T09:27:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (668 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (88 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (87 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:27:42 dist-test-slave-dfzm krb5kdc[5918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:27:42 dist-test-slave-dfzm krb5kdc[5918](info): set up 2 sockets
Sep 26 09:27:42 dist-test-slave-dfzm krb5kdc[5918](info): commencing operation
krb5kdc: starting...
W20260926 09:27:43.873832 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:27:43 dist-test-slave-dfzm krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:27:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:27:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:27:43.992831 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39451
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:44.074447 5931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:44.074707 5931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:44.074753 5931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:44.075981 5931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:27:44.076058 5931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:44.076081 5931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:44.076102 5931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:27:44.076131 5931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:27:44.077744 5931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:44.078032 5931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:44.078259 5931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:44.083370 5939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:44.083374 5937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:44.083375 5936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:44.083556 5931 server_base.cc:1034] running on GCE node
I20260926 09:27:44.083791 5931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:44.084090 5931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:44.085276 5931 hybrid_clock.cc:648] HybridClock initialized: now 1790414864085252 us; error 41 us; skew 500 ppm
Sep 26 09:27:44 dist-test-slave-dfzm krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:44.087815 5931 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:27:44.088243 5931 webserver.cc:460] Webserver started at http://127.0.98.62:43039/ using document root <none> and password file <none>
I20260926 09:27:44.088490 5931 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:44.088560 5931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:44.088707 5931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:44.089871 5931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "d8e2e34de66d47cd864e1cf0a9a2f9c3"
format_stamp: "Formatted at 2026-09-26 09:27:44 on dist-test-slave-dfzm"
server_key: "9dd5925d4c0e3db738328eea365f9c35"
server_key_iv: "2e9fda1d32414ed925502c953c1c24c3"
server_key_version: "encryptionkey@0"
I20260926 09:27:44.090229 5931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "d8e2e34de66d47cd864e1cf0a9a2f9c3"
format_stamp: "Formatted at 2026-09-26 09:27:44 on dist-test-slave-dfzm"
server_key: "9dd5925d4c0e3db738328eea365f9c35"
server_key_iv: "2e9fda1d32414ed925502c953c1c24c3"
server_key_version: "encryptionkey@0"
I20260926 09:27:44.091517 5931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:27:44.092401 5946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:44.092589 5931 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:44.092689 5931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "d8e2e34de66d47cd864e1cf0a9a2f9c3"
format_stamp: "Formatted at 2026-09-26 09:27:44 on dist-test-slave-dfzm"
server_key: "9dd5925d4c0e3db738328eea365f9c35"
server_key_iv: "2e9fda1d32414ed925502c953c1c24c3"
server_key_version: "encryptionkey@0"
I20260926 09:27:44.092772 5931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:44.111781 5931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:44.113790 5931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:44.113953 5931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:44.117930 5931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39451
I20260926 09:27:44.117970 5998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39451 every 8 connection(s)
I20260926 09:27:44.118507 5931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:27:44.119194 5999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:44.121515 5999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8e2e34de66d47cd864e1cf0a9a2f9c3: Bootstrap starting.
I20260926 09:27:44.122114 5999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8e2e34de66d47cd864e1cf0a9a2f9c3: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:44.122359 5999 log.cc:824] T 00000000000000000000000000000000 P d8e2e34de66d47cd864e1cf0a9a2f9c3: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:44.122992 5999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8e2e34de66d47cd864e1cf0a9a2f9c3: No bootstrap required, opened a new log
I20260926 09:27:44.124565 5999 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8e2e34de66d47cd864e1cf0a9a2f9c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e2e34de66d47cd864e1cf0a9a2f9c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39451 } }
I20260926 09:27:44.124702 5999 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8e2e34de66d47cd864e1cf0a9a2f9c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:44.124727 5999 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8e2e34de66d47cd864e1cf0a9a2f9c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8e2e34de66d47cd864e1cf0a9a2f9c3, State: Initialized, Role: FOLLOWER
I20260926 09:27:44.124836 5999 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8e2e34de66d47cd864e1cf0a9a2f9c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e2e34de66d47cd864e1cf0a9a2f9c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39451 } }
I20260926 09:27:44.124895 5999 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8e2e34de66d47cd864e1cf0a9a2f9c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:27:44.124915 5999 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8e2e34de66d47cd864e1cf0a9a2f9c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:27:44.124941 5999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8e2e34de66d47cd864e1cf0a9a2f9c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:44.125860 5999 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8e2e34de66d47cd864e1cf0a9a2f9c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e2e34de66d47cd864e1cf0a9a2f9c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39451 } }
I20260926 09:27:44.125947 5999 leader_election.cc:302] T 00000000000000000000000000000000 P d8e2e34de66d47cd864e1cf0a9a2f9c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8e2e34de66d47cd864e1cf0a9a2f9c3; no voters:
I20260926 09:27:44.126071 5999 leader_election.cc:288] T 00000000000000000000000000000000 P d8e2e34de66d47cd864e1cf0a9a2f9c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:27:44.126232 6003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8e2e34de66d47cd864e1cf0a9a2f9c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:44.126320 5999 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8e2e34de66d47cd864e1cf0a9a2f9c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:27:44.126368 6003 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8e2e34de66d47cd864e1cf0a9a2f9c3 [term 1 LEADER]: Becoming Leader. State: Replica: d8e2e34de66d47cd864e1cf0a9a2f9c3, State: Running, Role: LEADER
I20260926 09:27:44.126441 6003 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8e2e34de66d47cd864e1cf0a9a2f9c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e2e34de66d47cd864e1cf0a9a2f9c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39451 } }
I20260926 09:27:44.126814 6002 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e2e34de66d47cd864e1cf0a9a2f9c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8e2e34de66d47cd864e1cf0a9a2f9c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e2e34de66d47cd864e1cf0a9a2f9c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39451 } } }
I20260926 09:27:44.127046 6002 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e2e34de66d47cd864e1cf0a9a2f9c3 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:44.126977 6004 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e2e34de66d47cd864e1cf0a9a2f9c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8e2e34de66d47cd864e1cf0a9a2f9c3. Latest consensus state: current_term: 1 leader_uuid: "d8e2e34de66d47cd864e1cf0a9a2f9c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e2e34de66d47cd864e1cf0a9a2f9c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39451 } } }
I20260926 09:27:44.127223 6004 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e2e34de66d47cd864e1cf0a9a2f9c3 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:44.127547 6011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:27:44.127715 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 5931
I20260926 09:27:44.127825 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:27:44.128041 392 external_mini_cluster.cc:1428] Setting key b7ffb8776624179d1218a4c01c75b61f
I20260926 09:27:44.128201 6011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:27:44.129669 6011 catalog_manager.cc:1347] Generated new cluster ID: 1ca90e2d3b5e493898befa9c8ea283e0
I20260926 09:27:44.129721 6011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:27:44 dist-test-slave-dfzm krb5kdc[5918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:44.138181 6011 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:27:44.138752 6011 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:27:44.138801 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:44.128509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.0.98.62:39451)
0926 09:27:44.128673 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:44.128682 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:44.128957 (+ 275us) server_negotiation.cc:407] Connection header received
0926 09:27:44.129079 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:44.129087 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:44.129144 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:44.129292 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:44.130101 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:44.131006 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:44.131786 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:44.132227 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:44.132354 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:44.134813 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:44.134816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:44.134817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:44.134829 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:44.136632 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:44.137024 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:44.137027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:44.137028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:44.137132 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:44.137391 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:44.137392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:44.137393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:44.137739 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:44.137907 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:44.138430 (+ 523us) server_negotiation.cc:299] Negotiation successful
0926 09:27:44.138574 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
I20260926 09:27:44.142575 6011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8e2e34de66d47cd864e1cf0a9a2f9c3: Generated new TSK 0
I20260926 09:27:44.142794 6011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:27:44.168167 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39451
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:44.247932 6025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:44.248164 6025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:44.248204 6025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:44.249401 6025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:44.249469 6025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:44.249517 6025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:27:44.251017 6025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:44.251273 6025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:44.251502 6025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:44.257236 6030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:44.257234 6031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:44.257246 6033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:44.257503 6025 server_base.cc:1034] running on GCE node
I20260926 09:27:44.257748 6025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:44.257992 6025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:44.259115 6025 hybrid_clock.cc:648] HybridClock initialized: now 1790414864259100 us; error 31 us; skew 500 ppm
Sep 26 09:27:44 dist-test-slave-dfzm krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:44.261526 6025 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:27:44.262063 6025 webserver.cc:460] Webserver started at http://127.0.98.1:39035/ using document root <none> and password file <none>
I20260926 09:27:44.262286 6025 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:44.262331 6025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:44.262481 6025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:44.263531 6025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "223a9b452a7d4acf8c52f07cd1dae8dc"
format_stamp: "Formatted at 2026-09-26 09:27:44 on dist-test-slave-dfzm"
server_key: "061233fe40b8b866bee63cc358c4498d"
server_key_iv: "53d0026145061167787ba2eff8f52678"
server_key_version: "encryptionkey@0"
I20260926 09:27:44.263890 6025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "223a9b452a7d4acf8c52f07cd1dae8dc"
format_stamp: "Formatted at 2026-09-26 09:27:44 on dist-test-slave-dfzm"
server_key: "061233fe40b8b866bee63cc358c4498d"
server_key_iv: "53d0026145061167787ba2eff8f52678"
server_key_version: "encryptionkey@0"
I20260926 09:27:44.265198 6025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:27:44.265993 6040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:44.266160 6025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:44.266240 6025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "223a9b452a7d4acf8c52f07cd1dae8dc"
format_stamp: "Formatted at 2026-09-26 09:27:44 on dist-test-slave-dfzm"
server_key: "061233fe40b8b866bee63cc358c4498d"
server_key_iv: "53d0026145061167787ba2eff8f52678"
server_key_version: "encryptionkey@0"
I20260926 09:27:44.266292 6025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:44.282449 6025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:44.284920 6025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:44.285039 6025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:44.285243 6025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:44.285507 6025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:44.285537 6025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:44.285559 6025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:44.285571 6025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:44.291205 6025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37397
I20260926 09:27:44.291261 6153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37397 every 8 connection(s)
I20260926 09:27:44.291646 6025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:27:44.293025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 6025
I20260926 09:27:44.293157 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:27:44.293385 392 external_mini_cluster.cc:1428] Setting key 2c3819d46a92924c94cc16e972ee63a7
Sep 26 09:27:44 dist-test-slave-dfzm krb5kdc[5918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 09:27:44.302225 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:44.292420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55675 (local address 127.0.98.62:39451)
0926 09:27:44.292500 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:44.292502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:44.292714 (+ 212us) server_negotiation.cc:407] Connection header received
0926 09:27:44.293999 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:44.294000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:44.294011 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:44.294061 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:44.294752 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:44.295307 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:44.296407 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:44.296768 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:44.296839 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:44.299187 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:44.299189 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:44.299190 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:44.299201 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:44.300846 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:44.301249 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:44.301250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:44.301250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:44.301311 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:44.301548 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:44.301549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:44.301549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:44.301707 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:44.301779 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:44.302039 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 09:27:44.302131 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 09:27:44.302439 6156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:44.292404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39451 (local address 127.0.98.1:55675)
0926 09:27:44.292591 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:44.292627 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:44.293902 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:44.294155 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:44.294159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:44.294465 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:44.294654 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:44.294659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:44.295432 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:44.295433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:44.296313 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:44.296316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:44.296875 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:44.296876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:44.296987 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:44.297639 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:44.297649 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:44.299087 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:44.300940 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:44.300946 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:44.300948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:44.301168 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:44.301384 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:44.301385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:44.301385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:44.301481 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:44.301775 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:44.301779 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:44.301898 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 09:27:44.302015 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 09:27:44.302172 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":92,"threads_started":1}
I20260926 09:27:44.302930 6154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39451
I20260926 09:27:44.303049 6154 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:44.303232 6154 heartbeater.cc:511] Master 127.0.98.62:39451 requested a full tablet report, sending...
I20260926 09:27:44.303810 5963 ts_manager.cc:194] Registered new tserver with Master: 223a9b452a7d4acf8c52f07cd1dae8dc (127.0.98.1:37397)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 09:27:44.304489 5963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:27:44.320524 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39451
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:44.403337 6159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:44.403594 6159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:44.403637 6159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:44.404875 6159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:44.404956 6159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:44.405004 6159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:27:44.406584 6159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:44.406854 6159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:44.407087 6159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:27:44.412148 6159 server_base.cc:1034] running on GCE node
W20260926 09:27:44.412108 6167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:44.412120 6165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:44.412124 6164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:44.412617 6159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:44.412830 6159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:44.413970 6159 hybrid_clock.cc:648] HybridClock initialized: now 1790414864413963 us; error 30 us; skew 500 ppm
Sep 26 09:27:44 dist-test-slave-dfzm krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:44.416430 6159 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:27:44.416863 6159 webserver.cc:460] Webserver started at http://127.0.98.2:42583/ using document root <none> and password file <none>
I20260926 09:27:44.417058 6159 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:44.417100 6159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:44.417189 6159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:44.418259 6159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "966631a5a57e43028c8a6dfc4cc18d07"
format_stamp: "Formatted at 2026-09-26 09:27:44 on dist-test-slave-dfzm"
server_key: "77f9583c65314d4f5bf91e26ede540bb"
server_key_iv: "9878ac24384dc74ef402665db776bee2"
server_key_version: "encryptionkey@0"
I20260926 09:27:44.418569 6159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "966631a5a57e43028c8a6dfc4cc18d07"
format_stamp: "Formatted at 2026-09-26 09:27:44 on dist-test-slave-dfzm"
server_key: "77f9583c65314d4f5bf91e26ede540bb"
server_key_iv: "9878ac24384dc74ef402665db776bee2"
server_key_version: "encryptionkey@0"
I20260926 09:27:44.419898 6159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:27:44.420650 6174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:44.420969 6159 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:44.421082 6159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "966631a5a57e43028c8a6dfc4cc18d07"
format_stamp: "Formatted at 2026-09-26 09:27:44 on dist-test-slave-dfzm"
server_key: "77f9583c65314d4f5bf91e26ede540bb"
server_key_iv: "9878ac24384dc74ef402665db776bee2"
server_key_version: "encryptionkey@0"
I20260926 09:27:44.421164 6159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:44.440076 6159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:44.442325 6159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:44.442587 6159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:44.442847 6159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:44.443169 6159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:44.443203 6159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:44.443226 6159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:44.443276 6159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:44.448899 6159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42841
I20260926 09:27:44.449007 6287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42841 every 8 connection(s)
I20260926 09:27:44.449361 6159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:27:44 dist-test-slave-dfzm krb5kdc[5918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:44.455559 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 6159
I20260926 09:27:44.455657 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:27:44.455866 392 external_mini_cluster.cc:1428] Setting key 5dd372164f1b676571d3340cc7cf6a91
I20260926 09:27:44.459002 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:44.450179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41709 (local address 127.0.98.62:39451)
0926 09:27:44.450224 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:44.450226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:44.450444 (+ 218us) server_negotiation.cc:407] Connection header received
0926 09:27:44.451417 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:44.451418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:44.451430 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:44.451483 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:44.452133 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:44.452633 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:44.453681 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:44.453886 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:44.453962 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:44.456181 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:44.456183 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:44.456183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:44.456191 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:44.457730 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:44.458138 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:44.458139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:44.458140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:44.458206 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:44.458414 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:44.458415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:44.458415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:44.458557 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:44.458638 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:44.458866 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 09:27:44.458938 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:27:44.459147 6290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:44.450150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39451 (local address 127.0.98.2:41709)
0926 09:27:44.450292 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:44.450344 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:44.451291 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:44.451564 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:44.451568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:44.451834 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:44.452026 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:44.452030 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:44.452725 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:44.452727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:44.453537 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:44.453539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:44.453980 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:44.453981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:44.454109 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:44.454602 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:44.454610 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:44.456033 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:44.457843 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:44.457854 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:44.457856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:44.458067 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:44.458270 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:44.458270 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:44.458271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:44.458357 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:44.458633 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:44.458637 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:44.458750 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 09:27:44.458836 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 09:27:44.458953 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
I20260926 09:27:44.459643 6288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39451
I20260926 09:27:44.459738 6288 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:44.459932 6288 heartbeater.cc:511] Master 127.0.98.62:39451 requested a full tablet report, sending...
I20260926 09:27:44.460289 5963 ts_manager.cc:194] Registered new tserver with Master: 966631a5a57e43028c8a6dfc4cc18d07 (127.0.98.2:42841)
I20260926 09:27:44.460659 5963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41709
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:27:44.480628 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39451
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:44.561766 6293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:44.561969 6293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:44.561993 6293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:44.563078 6293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:44.563112 6293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:44.563136 6293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:27:44.564551 6293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:44.564793 6293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:44.564981 6293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:44.569936 6298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:44.569945 6301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:44.569947 6299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:44.570137 6293 server_base.cc:1034] running on GCE node
I20260926 09:27:44.570363 6293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:44.570618 6293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:44.571756 6293 hybrid_clock.cc:648] HybridClock initialized: now 1790414864571738 us; error 34 us; skew 500 ppm
Sep 26 09:27:44 dist-test-slave-dfzm krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:44.574311 6293 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:27:44.574777 6293 webserver.cc:460] Webserver started at http://127.0.98.3:40173/ using document root <none> and password file <none>
I20260926 09:27:44.575057 6293 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:44.575104 6293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:44.575193 6293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:44.576175 6293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "b649327bd39f458599b98484a0e27af7"
format_stamp: "Formatted at 2026-09-26 09:27:44 on dist-test-slave-dfzm"
server_key: "bfe825e67050fe9a40f0384589294a59"
server_key_iv: "0019a16458f7cb1fe5f6d032fca9da0a"
server_key_version: "encryptionkey@0"
I20260926 09:27:44.576483 6293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b649327bd39f458599b98484a0e27af7"
format_stamp: "Formatted at 2026-09-26 09:27:44 on dist-test-slave-dfzm"
server_key: "bfe825e67050fe9a40f0384589294a59"
server_key_iv: "0019a16458f7cb1fe5f6d032fca9da0a"
server_key_version: "encryptionkey@0"
I20260926 09:27:44.577721 6293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:27:44.578419 6308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:44.578586 6293 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:44.578668 6293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "b649327bd39f458599b98484a0e27af7"
format_stamp: "Formatted at 2026-09-26 09:27:44 on dist-test-slave-dfzm"
server_key: "bfe825e67050fe9a40f0384589294a59"
server_key_iv: "0019a16458f7cb1fe5f6d032fca9da0a"
server_key_version: "encryptionkey@0"
I20260926 09:27:44.578719 6293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:44.586479 6293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:44.588248 6293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:44.588358 6293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:44.588529 6293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:44.588845 6293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:44.588883 6293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:44.588913 6293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:44.588964 6293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:44.596481 6293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45665
I20260926 09:27:44.596510 6421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45665 every 8 connection(s)
I20260926 09:27:44.596956 6293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:27:44 dist-test-slave-dfzm krb5kdc[5918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:44.605432 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 6293
I20260926 09:27:44.605540 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:27:44.605778 392 external_mini_cluster.cc:1428] Setting key 95c20fcc5a7ad4b06ada126fa3036073
I20260926 09:27:44.605811 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:44.598011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38329 (local address 127.0.98.62:39451)
0926 09:27:44.598074 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:44.598076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:44.598087 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:27:44.598908 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:44.598909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:44.598917 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:44.598967 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:44.599548 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:44.600027 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:44.600980 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:44.601158 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:44.601210 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:44.603312 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:44.603313 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:44.603313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:44.603320 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:44.604533 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:44.604881 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:44.604882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:44.604883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:44.604930 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:44.605132 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:44.605132 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:44.605132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:44.605246 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:44.605296 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:44.605681 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 09:27:44.605745 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 09:27:44.606099 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:44.597712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39451 (local address 127.0.98.3:38329)
0926 09:27:44.597868 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:44.597911 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:44.598814 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:44.599031 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:44.599034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:44.599286 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:44.599481 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:44.599485 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:44.600105 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:44.600106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:44.600904 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:44.600906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:44.601229 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:44.601230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:44.601326 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:44.601864 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:44.601873 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:44.603225 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:44.604605 (+ 1380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:44.604610 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:44.604612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:44.604817 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:44.604987 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:44.604988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:44.604988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:44.605079 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:44.605318 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:44.605324 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:44.605490 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 09:27:44.605656 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 09:27:44.605832 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20260926 09:27:44.606581 6422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39451
I20260926 09:27:44.606703 6422 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:44.606881 6422 heartbeater.cc:511] Master 127.0.98.62:39451 requested a full tablet report, sending...
I20260926 09:27:44.607278 5963 ts_manager.cc:194] Registered new tserver with Master: b649327bd39f458599b98484a0e27af7 (127.0.98.3:45665)
I20260926 09:27:44.607614 5963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38329
I20260926 09:27:44.617659 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/test-admin.keytab.
Sep 26 09:27:44 dist-test-slave-dfzm krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:44.633484 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 09:27:44.635174 6429 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:44.634364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59518 (local address 127.0.98.1:37397)
0926 09:27:44.634537 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:44.634543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:44.634560 (+ 17us) server_negotiation.cc:407] Connection header received
0926 09:27:44.634585 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:44.634587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:44.634606 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:44.634728 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:44.635073 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59518: BlockingRecv error: recv got EOF from 127.0.0.1:59518 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
W20260926 09:27:44.635383 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/krb5cc)
Sep 26 09:27:44 dist-test-slave-dfzm krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:44.636441 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 09:27:44 dist-test-slave-dfzm krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:44.637375 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 09:27:44 dist-test-slave-dfzm krb5kdc[5918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 09:27:44.645699 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:44.638101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59526 (local address 127.0.98.1:37397)
0926 09:27:44.638157 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:44.638159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:44.638275 (+ 116us) server_negotiation.cc:407] Connection header received
0926 09:27:44.638302 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:44.638303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:44.638314 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:44.638361 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:44.638939 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:44.639651 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:44.640342 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:44.640647 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:44.640732 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:44.642876 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:44.642883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:44.642883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:44.642890 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:44.644646 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:44.644974 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:44.644977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:44.644977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:44.645031 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:44.645198 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:44.645199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:44.645199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:44.645336 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:44.645447 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:44.645502 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 09:27:44.645562 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 09:27:44.647528 6429 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:44.646993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59532 (local address 127.0.98.1:37397)
0926 09:27:44.647049 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:44.647051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:44.647127 (+ 76us) server_negotiation.cc:407] Connection header received
0926 09:27:44.647158 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:44.647159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:44.647167 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:44.647215 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:44.647461 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59532: BlockingRecv error: recv got EOF from 127.0.0.1:59532 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 09:27:44.647634 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 09:27:44 dist-test-slave-dfzm krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:44.648555 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 09:27:44 dist-test-slave-dfzm krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:44.649358 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 09:27:44 dist-test-slave-dfzm krb5kdc[5918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 09:27:44.656188 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:44.649956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59546 (local address 127.0.98.1:37397)
0926 09:27:44.650015 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:44.650017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:44.650061 (+ 44us) server_negotiation.cc:407] Connection header received
0926 09:27:44.650121 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:44.650122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:44.650131 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:44.650179 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:44.650544 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:44.651021 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:44.651689 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:44.651894 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:44.651975 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:44.653983 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:44.653984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:44.653984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:44.653990 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:44.655123 (+ 1133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:44.655460 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:44.655461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:44.655462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:44.655516 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:44.655711 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:44.655712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:44.655712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:44.655806 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:44.655908 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:44.656018 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 09:27:44.656091 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 09:27:44.657675 6429 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:44.657120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59556 (local address 127.0.98.1:37397)
0926 09:27:44.657179 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:44.657181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:44.657221 (+ 40us) server_negotiation.cc:407] Connection header received
0926 09:27:44.657290 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:44.657291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:44.657300 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:44.657353 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:44.657631 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59556: BlockingRecv error: recv got EOF from 127.0.0.1:59556 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 09:27:44.657788 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414839069479-392-0/krb5kdc/krb5cc)
Sep 26 09:27:44 dist-test-slave-dfzm krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:44.658661 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 09:27:44 dist-test-slave-dfzm krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:44.659483 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 09:27:44 dist-test-slave-dfzm krb5kdc[5918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 09:27:44.666024 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:44.659968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59568 (local address 127.0.98.1:37397)
0926 09:27:44.660003 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:44.660010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:44.660038 (+ 28us) server_negotiation.cc:407] Connection header received
0926 09:27:44.660087 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:44.660087 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:44.660094 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:44.660141 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:44.660466 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:44.660915 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:44.661572 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:44.661787 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:44.661837 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:44.663730 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:44.663731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:44.663731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:44.663737 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:44.665086 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:44.665357 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:44.665358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:44.665358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:44.665416 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:44.665570 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:44.665571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:44.665571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:44.665722 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:44.665792 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:44.665903 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 09:27:44.665956 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 09:27:44.666757 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 6025
I20260926 09:27:44.671139 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 6159
I20260926 09:27:44.675549 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 6293
I20260926 09:27:44.679981 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 5931
2026-09-26T09:27:44Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1858 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:27:44 dist-test-slave-dfzm krb5kdc[6442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:27:44 dist-test-slave-dfzm krb5kdc[6442](info): set up 2 sockets
Sep 26 09:27:44 dist-test-slave-dfzm krb5kdc[6442](info): commencing operation
krb5kdc: starting...
W20260926 09:27:45.737901 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:27:45 dist-test-slave-dfzm krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:27:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:27:45Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 09:27:45.856165 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39437
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39437
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:45.936626 6455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:45.936823 6455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:45.936846 6455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:45.938069 6455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:27:45.938112 6455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:45.938126 6455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:45.938139 6455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:27:45.938182 6455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:27:45.939628 6455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:45.939864 6455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:45.940096 6455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:45.945070 6461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:45.945106 6463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:45.945250 6455 server_base.cc:1034] running on GCE node
W20260926 09:27:45.945076 6460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:45.945541 6455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:45.945866 6455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:45.947038 6455 hybrid_clock.cc:648] HybridClock initialized: now 1790414865947026 us; error 30 us; skew 500 ppm
Sep 26 09:27:45 dist-test-slave-dfzm krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414865, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:45.949470 6455 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 09:27:45.949920 6455 webserver.cc:460] Webserver started at http://127.0.98.62:45941/ using document root <none> and password file <none>
I20260926 09:27:45.950126 6455 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:45.950168 6455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:45.950256 6455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:45.951252 6455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "705253badbc740e49445bb9ab7943235"
format_stamp: "Formatted at 2026-09-26 09:27:45 on dist-test-slave-dfzm"
server_key: "6bdd9efcbfbf12a6c644c3839c3f5050"
server_key_iv: "589aba16d4ba788fbd6a72fd08e11330"
server_key_version: "encryptionkey@0"
I20260926 09:27:45.951594 6455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "705253badbc740e49445bb9ab7943235"
format_stamp: "Formatted at 2026-09-26 09:27:45 on dist-test-slave-dfzm"
server_key: "6bdd9efcbfbf12a6c644c3839c3f5050"
server_key_iv: "589aba16d4ba788fbd6a72fd08e11330"
server_key_version: "encryptionkey@0"
I20260926 09:27:45.952862 6455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:27:45.953707 6470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:45.953934 6455 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:45.954035 6455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "705253badbc740e49445bb9ab7943235"
format_stamp: "Formatted at 2026-09-26 09:27:45 on dist-test-slave-dfzm"
server_key: "6bdd9efcbfbf12a6c644c3839c3f5050"
server_key_iv: "589aba16d4ba788fbd6a72fd08e11330"
server_key_version: "encryptionkey@0"
I20260926 09:27:45.954118 6455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:45.973067 6455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:45.975759 6455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:45.975883 6455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:45.979961 6455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39437
I20260926 09:27:45.980065 6522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39437 every 8 connection(s)
I20260926 09:27:45.980400 6455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:27:45.981017 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 6455
I20260926 09:27:45.981132 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:27:45.981370 392 external_mini_cluster.cc:1428] Setting key 41f7b4d69595388cec6ee9a9b6157a7a
I20260926 09:27:45.981761 6523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:45.985314 6523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235: Bootstrap starting.
I20260926 09:27:45.986066 6523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:45.986378 6523 log.cc:824] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:45.987141 6523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235: No bootstrap required, opened a new log
Sep 26 09:27:45 dist-test-slave-dfzm krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:27:45.989547 6523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } }
I20260926 09:27:45.989766 6523 raft_consensus.cc:374] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:45.989813 6523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 705253badbc740e49445bb9ab7943235, State: Initialized, Role: FOLLOWER
I20260926 09:27:45.989991 6523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } }
I20260926 09:27:45.990084 6523 raft_consensus.cc:388] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:27:45.990123 6523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:27:45.990163 6523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:45.990798 6523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } }
I20260926 09:27:45.990907 6523 leader_election.cc:302] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 705253badbc740e49445bb9ab7943235; no voters:
I20260926 09:27:45.991107 6523 leader_election.cc:288] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:27:45.991135 6528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:45.991396 6523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:27:45.991719 6528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [term 1 LEADER]: Becoming Leader. State: Replica: 705253badbc740e49445bb9ab7943235, State: Running, Role: LEADER
I20260926 09:27:45.991858 6528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } }
I20260926 09:27:45.992328 6528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 705253badbc740e49445bb9ab7943235. Latest consensus state: current_term: 1 leader_uuid: "705253badbc740e49445bb9ab7943235" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } }
I20260926 09:27:45.992455 6528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:45.992734 6540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:27:45.993299 6540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:27:45.994403 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:45.982297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55632 (local address 127.0.98.62:39437)
0926 09:27:45.982482 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:45.982490 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:45.982514 (+ 24us) server_negotiation.cc:407] Connection header received
0926 09:27:45.983114 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:45.983122 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:45.983205 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:45.983363 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:45.984144 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:45.985302 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:45.986424 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:45.986955 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:45.987040 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:45.990170 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:45.990174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:45.990174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:45.990187 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:45.992529 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:45.992968 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:45.992971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:45.992971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:45.993077 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:45.993271 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:45.993272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:45.993272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:45.993630 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:45.993775 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:45.994065 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 09:27:45.994213 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260926 09:27:45.996104 6540 catalog_manager.cc:1347] Generated new cluster ID: 00db4655e98e4497b0dbe52a74a75983
I20260926 09:27:45.996162 6540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:27:45.997841 6529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "705253badbc740e49445bb9ab7943235" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } }
I20260926 09:27:45.997938 6529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:46.004316 6540 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:27:46.004992 6540 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:27:46.013963 6540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235: Generated new TSK 0
I20260926 09:27:46.014197 6540 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.1.keytab.
W20260926 09:27:46.152755 392 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.0.98.1: real 0.129s user 0.000s sys 0.003s
I20260926 09:27:46.153132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39437
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:46.234936 6549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:46.235167 6549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:46.235195 6549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:46.236327 6549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:46.236375 6549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:46.236406 6549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:27:46.237828 6549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:46.238024 6549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:46.238209 6549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:46.243456 6554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:46.243481 6557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:46.243432 6555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:46.243669 6549 server_base.cc:1034] running on GCE node
I20260926 09:27:46.243852 6549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:46.244051 6549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:46.245201 6549 hybrid_clock.cc:648] HybridClock initialized: now 1790414866245188 us; error 30 us; skew 500 ppm
Sep 26 09:27:46 dist-test-slave-dfzm krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414866, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:46.247920 6549 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 09:27:46.248446 6549 webserver.cc:460] Webserver started at http://127.0.98.1:40889/ using document root <none> and password file <none>
I20260926 09:27:46.248675 6549 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:46.248746 6549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:46.248899 6549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:46.250049 6549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "d7110ef2e34945f9958fbbcfec4c6f4d"
format_stamp: "Formatted at 2026-09-26 09:27:46 on dist-test-slave-dfzm"
server_key: "408508d91002cf4e43590d285e00ead4"
server_key_iv: "7cf08a35f4c046c3e3449f2f61403098"
server_key_version: "encryptionkey@0"
I20260926 09:27:46.250408 6549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d7110ef2e34945f9958fbbcfec4c6f4d"
format_stamp: "Formatted at 2026-09-26 09:27:46 on dist-test-slave-dfzm"
server_key: "408508d91002cf4e43590d285e00ead4"
server_key_iv: "7cf08a35f4c046c3e3449f2f61403098"
server_key_version: "encryptionkey@0"
I20260926 09:27:46.251716 6549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:27:46.252408 6564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:46.252607 6549 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:46.252734 6549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "d7110ef2e34945f9958fbbcfec4c6f4d"
format_stamp: "Formatted at 2026-09-26 09:27:46 on dist-test-slave-dfzm"
server_key: "408508d91002cf4e43590d285e00ead4"
server_key_iv: "7cf08a35f4c046c3e3449f2f61403098"
server_key_version: "encryptionkey@0"
I20260926 09:27:46.252810 6549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:46.273036 6549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:46.275524 6549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:46.275698 6549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:46.275944 6549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:46.276283 6549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:46.276340 6549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:46.276386 6549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:46.276410 6549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:46.282321 6549 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44919
I20260926 09:27:46.282404 6677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44919 every 8 connection(s)
I20260926 09:27:46.282796 6549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:27:46.287935 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 6549
I20260926 09:27:46.288125 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:27:46.288336 392 external_mini_cluster.cc:1428] Setting key 6aaf22f33a28e56469732702742ac0fe
Sep 26 09:27:46 dist-test-slave-dfzm krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414866, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:27:46.293090 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:46.283643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35715 (local address 127.0.98.62:39437)
0926 09:27:46.283723 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:46.283726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:46.283944 (+ 218us) server_negotiation.cc:407] Connection header received
0926 09:27:46.284467 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:46.284468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:46.284479 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:46.284537 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:46.285533 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.286085 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.287127 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.287349 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.287428 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:46.289988 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:46.289989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:46.289989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:46.289997 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:46.291511 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:46.292038 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:46.292039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:46.292040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:46.292107 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:46.292360 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:46.292361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:46.292361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:46.292498 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:46.292604 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:46.292901 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 09:27:46.292992 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 09:27:46.293293 6680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:46.283582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.98.1:35715)
0926 09:27:46.283794 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:46.283839 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:46.284351 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:46.284630 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:46.284633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:46.285206 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:46.285412 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:46.285417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.286189 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.286190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:46.287018 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:46.287020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.287465 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.287466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:46.287582 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:46.288268 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:46.288278 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:46.289886 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:46.291590 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:46.291596 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:46.291598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:46.291898 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:46.292175 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:46.292176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:46.292176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:46.292285 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:46.292623 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:46.292629 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:46.292761 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 09:27:46.292881 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 09:27:46.293058 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":90,"threads_started":1}
I20260926 09:27:46.293877 6678 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39437
I20260926 09:27:46.293970 6678 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:46.294168 6678 heartbeater.cc:511] Master 127.0.98.62:39437 requested a full tablet report, sending...
I20260926 09:27:46.294739 6487 ts_manager.cc:194] Registered new tserver with Master: d7110ef2e34945f9958fbbcfec4c6f4d (127.0.98.1:44919)
I20260926 09:27:46.295341 6487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35715
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 09:27:46.315156 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39437
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:46.398360 6683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:46.398571 6683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:46.398597 6683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:46.399750 6683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:46.399791 6683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:46.399856 6683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:27:46.401343 6683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:46.401535 6683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:46.401778 6683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:46.406767 6688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:46.406785 6689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:46.406833 6691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:46.407099 6683 server_base.cc:1034] running on GCE node
I20260926 09:27:46.407279 6683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:46.407490 6683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:46.408674 6683 hybrid_clock.cc:648] HybridClock initialized: now 1790414866408654 us; error 46 us; skew 500 ppm
Sep 26 09:27:46 dist-test-slave-dfzm krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414866, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:46.411396 6683 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 09:27:46.411860 6683 webserver.cc:460] Webserver started at http://127.0.98.2:37627/ using document root <none> and password file <none>
I20260926 09:27:46.412106 6683 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:46.412179 6683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:46.412326 6683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:46.413374 6683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "4094f93fa463479dbaed35e8f6e5a437"
format_stamp: "Formatted at 2026-09-26 09:27:46 on dist-test-slave-dfzm"
server_key: "fde31024897de9e983a8667d1b1c9886"
server_key_iv: "4174bbf8cf7e217224c8f630b2b23240"
server_key_version: "encryptionkey@0"
I20260926 09:27:46.413769 6683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4094f93fa463479dbaed35e8f6e5a437"
format_stamp: "Formatted at 2026-09-26 09:27:46 on dist-test-slave-dfzm"
server_key: "fde31024897de9e983a8667d1b1c9886"
server_key_iv: "4174bbf8cf7e217224c8f630b2b23240"
server_key_version: "encryptionkey@0"
I20260926 09:27:46.415126 6683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:27:46.415961 6698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:46.416133 6683 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:27:46.416245 6683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "4094f93fa463479dbaed35e8f6e5a437"
format_stamp: "Formatted at 2026-09-26 09:27:46 on dist-test-slave-dfzm"
server_key: "fde31024897de9e983a8667d1b1c9886"
server_key_iv: "4174bbf8cf7e217224c8f630b2b23240"
server_key_version: "encryptionkey@0"
I20260926 09:27:46.416334 6683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:46.423252 6683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:46.425276 6683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:46.425457 6683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:46.425761 6683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:46.426088 6683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:46.426151 6683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:46.426192 6683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:46.426221 6683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:46.431900 6683 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45101
I20260926 09:27:46.431941 6811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45101 every 8 connection(s)
I20260926 09:27:46.432277 6683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:27:46 dist-test-slave-dfzm krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414866, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:27:46.440069 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 6683
I20260926 09:27:46.440210 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:27:46.440409 392 external_mini_cluster.cc:1428] Setting key d7c93a0ea357c3c3a9824c573136b2ac
I20260926 09:27:46.441994 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:46.433037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40867 (local address 127.0.98.62:39437)
0926 09:27:46.433098 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:46.433100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:46.433315 (+ 215us) server_negotiation.cc:407] Connection header received
0926 09:27:46.433930 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:46.433931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:46.433944 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:46.434000 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:46.434960 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.435459 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.436480 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.436676 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.436752 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:46.438928 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:46.438930 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:46.438930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:46.438937 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:46.440412 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:46.440916 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:46.440917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:46.440918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:46.440988 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:46.441289 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:46.441290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:46.441290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:46.441410 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:46.441492 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:46.441826 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 09:27:46.441905 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 09:27:46.442274 6814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:46.433026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.98.2:40867)
0926 09:27:46.433196 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:46.433231 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:46.433813 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:46.434083 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:46.434086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:46.434659 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:46.434859 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:46.434864 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.435564 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.435565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:46.436377 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:46.436379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.436757 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.436758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:46.436887 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:46.437418 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:46.437426 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:46.438834 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:46.440523 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:46.440530 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:46.440532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:46.440847 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:46.441064 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:46.441065 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:46.441066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:46.441199 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:46.441490 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:46.441497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:46.441691 (+ 194us) client_negotiation.cc:769] Sending connection context
0926 09:27:46.441794 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 09:27:46.441960 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260926 09:27:46.442797 6812 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39437
I20260926 09:27:46.442902 6812 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:46.443107 6812 heartbeater.cc:511] Master 127.0.98.62:39437 requested a full tablet report, sending...
I20260926 09:27:46.443588 6487 ts_manager.cc:194] Registered new tserver with Master: 4094f93fa463479dbaed35e8f6e5a437 (127.0.98.2:45101)
I20260926 09:27:46.443986 6487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40867
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 09:27:46.466754 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39437
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:46.546316 6817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:46.546630 6817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:46.546666 6817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:46.548097 6817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:46.548170 6817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:46.548214 6817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:27:46.549726 6817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:46.549988 6817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:46.550246 6817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:46.555209 6823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:46.555197 6825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:46.555197 6822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:46.555526 6817 server_base.cc:1034] running on GCE node
I20260926 09:27:46.555737 6817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:46.556052 6817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:46.557209 6817 hybrid_clock.cc:648] HybridClock initialized: now 1790414866557196 us; error 27 us; skew 500 ppm
Sep 26 09:27:46 dist-test-slave-dfzm krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414866, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:46.559846 6817 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 09:27:46.560348 6817 webserver.cc:460] Webserver started at http://127.0.98.3:33597/ using document root <none> and password file <none>
I20260926 09:27:46.560608 6817 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:46.560655 6817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:46.560828 6817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:46.561946 6817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "a9f71727a3f743c2b04bc0b854fdbad6"
format_stamp: "Formatted at 2026-09-26 09:27:46 on dist-test-slave-dfzm"
server_key: "9aba6fd1f95f5a7608c68234d1f27435"
server_key_iv: "397a4246be307ab0fd3dea3e0f1fba58"
server_key_version: "encryptionkey@0"
I20260926 09:27:46.562296 6817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a9f71727a3f743c2b04bc0b854fdbad6"
format_stamp: "Formatted at 2026-09-26 09:27:46 on dist-test-slave-dfzm"
server_key: "9aba6fd1f95f5a7608c68234d1f27435"
server_key_iv: "397a4246be307ab0fd3dea3e0f1fba58"
server_key_version: "encryptionkey@0"
I20260926 09:27:46.563527 6817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:46.564190 6832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:46.564349 6817 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:46.564452 6817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "a9f71727a3f743c2b04bc0b854fdbad6"
format_stamp: "Formatted at 2026-09-26 09:27:46 on dist-test-slave-dfzm"
server_key: "9aba6fd1f95f5a7608c68234d1f27435"
server_key_iv: "397a4246be307ab0fd3dea3e0f1fba58"
server_key_version: "encryptionkey@0"
I20260926 09:27:46.564524 6817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:46.574385 6817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:46.576247 6817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:46.576406 6817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:46.576623 6817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:46.576922 6817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:46.576977 6817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:46.577018 6817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:46.577047 6817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:46.582870 6817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36285
I20260926 09:27:46.582916 6945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36285 every 8 connection(s)
I20260926 09:27:46.583212 6817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:27:46 dist-test-slave-dfzm krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414866, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:27:46.591250 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 6817
I20260926 09:27:46.591365 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:27:46.591593 392 external_mini_cluster.cc:1428] Setting key b09045fbd375705c22eca81efbd85e1f
I20260926 09:27:46.592933 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:46.583907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60409 (local address 127.0.98.62:39437)
0926 09:27:46.584027 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:46.584029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:46.584238 (+ 209us) server_negotiation.cc:407] Connection header received
0926 09:27:46.584931 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:46.584933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:46.584946 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:46.585014 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:46.586097 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.586582 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.587668 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.587852 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.587902 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:46.590114 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:46.590115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:46.590115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:46.590121 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:46.591401 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:46.591884 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:46.591886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:46.591886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:46.591949 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:46.592280 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:46.592281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:46.592281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:46.592424 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:46.592495 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:46.592810 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 09:27:46.592871 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:27:46.593107 6948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:46.583824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.98.3:60409)
0926 09:27:46.584085 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:46.584135 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:46.584741 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:46.585141 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:46.585146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:46.585806 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:46.586013 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:46.586018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.586673 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.586674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:46.587581 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:46.587584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.587926 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.587927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:46.588045 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:46.588632 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:46.588641 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:46.590030 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:46.591478 (+ 1448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:46.591485 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:46.591487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:46.591828 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:46.592042 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:46.592043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:46.592044 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:46.592188 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:46.592499 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:46.592505 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:46.592688 (+ 183us) client_negotiation.cc:769] Sending connection context
0926 09:27:46.592791 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 09:27:46.592912 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":85,"threads_started":1}
I20260926 09:27:46.593547 6946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39437
I20260926 09:27:46.593677 6946 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:46.593851 6946 heartbeater.cc:511] Master 127.0.98.62:39437 requested a full tablet report, sending...
I20260926 09:27:46.594244 6487 ts_manager.cc:194] Registered new tserver with Master: a9f71727a3f743c2b04bc0b854fdbad6 (127.0.98.3:36285)
I20260926 09:27:46.594624 6487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60409
I20260926 09:27:46.603410 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260926 09:27:46.628322 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38949
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39437,127.0.98.61:38949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:46.708948 6951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:46.709167 6951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:46.709192 6951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:46.710395 6951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:27:46.710441 6951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:46.710455 6951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:46.710469 6951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:27:46.710480 6951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:27:46.711896 6951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39437,127.0.98.61:38949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
W20260926 09:27:46.711999 6951 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:39437,127.0.98.61:38949'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 09:27:46.712118 6951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:46.712288 6951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:46.717165 6956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:46.717228 6957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:46.717233 6959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:46.717392 6951 server_base.cc:1034] running on GCE node
I20260926 09:27:46.717676 6951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:46.717945 6951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:46.719082 6951 hybrid_clock.cc:648] HybridClock initialized: now 1790414866719071 us; error 29 us; skew 500 ppm
Sep 26 09:27:46 dist-test-slave-dfzm krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414866, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:46.721725 6951 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 09:27:46.722199 6951 webserver.cc:460] Webserver started at http://127.0.98.61:33873/ using document root <none> and password file <none>
I20260926 09:27:46.722441 6951 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:46.722529 6951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:46.722694 6951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:46.723766 6951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/data/instance:
uuid: "c056ade4d2f748619998a5e38bf29134"
format_stamp: "Formatted at 2026-09-26 09:27:46 on dist-test-slave-dfzm"
server_key: "7eb333d93122e062791d6df5155e1e33"
server_key_iv: "bfc6b8fea6c2b15caa33fae0029ddf9e"
server_key_version: "encryptionkey@0"
I20260926 09:27:46.724138 6951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/wal/instance:
uuid: "c056ade4d2f748619998a5e38bf29134"
format_stamp: "Formatted at 2026-09-26 09:27:46 on dist-test-slave-dfzm"
server_key: "7eb333d93122e062791d6df5155e1e33"
server_key_iv: "bfc6b8fea6c2b15caa33fae0029ddf9e"
server_key_version: "encryptionkey@0"
I20260926 09:27:46.725493 6951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260926 09:27:46.726282 6966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:46.726457 6951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:27:46.726557 6951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/wal
uuid: "c056ade4d2f748619998a5e38bf29134"
format_stamp: "Formatted at 2026-09-26 09:27:46 on dist-test-slave-dfzm"
server_key: "7eb333d93122e062791d6df5155e1e33"
server_key_iv: "bfc6b8fea6c2b15caa33fae0029ddf9e"
server_key_version: "encryptionkey@0"
I20260926 09:27:46.726657 6951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:46.737155 6951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:46.739084 6951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:46.739249 6951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:46.743148 6951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38949
I20260926 09:27:46.743176 7018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38949 every 8 connection(s)
I20260926 09:27:46.743533 6951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/data/info.pb
I20260926 09:27:46.744539 7019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:46.746126 7019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:27:46 dist-test-slave-dfzm krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414866, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:27:46.753034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 6951
I20260926 09:27:46.755923 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:46.746911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55638 (local address 127.0.98.62:39437)
0926 09:27:46.747011 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:46.747013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:46.747275 (+ 262us) server_negotiation.cc:407] Connection header received
0926 09:27:46.747857 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:46.747858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:46.747871 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:46.747928 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:46.748553 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.749055 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.750196 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.750403 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.750492 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:46.752640 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:46.752641 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:46.752642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:46.752657 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:46.754429 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:46.754898 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:46.754899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:46.754900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:46.754967 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:46.755211 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:46.755212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:46.755212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:46.755428 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:46.755497 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:46.755774 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 09:27:46.755858 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 09:27:46.756151 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:46.746894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.0.1:55638)
0926 09:27:46.747105 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:46.747174 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:46.747741 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:46.748005 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:46.748009 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:46.748261 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:46.748455 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:46.748458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.749161 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.749162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:46.750040 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:46.750042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.750500 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.750501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:46.750628 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:46.751109 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:46.751117 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:46.752544 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:46.754510 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:46.754517 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:46.754519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:46.754824 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:46.755037 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:46.755038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:46.755038 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:46.755145 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:46.755494 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:46.755499 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:46.755629 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 09:27:46.755721 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 09:27:46.755915 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":101,"threads_started":1}
I20260926 09:27:46.756800 7019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:27:46 dist-test-slave-dfzm krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 09:27:46 dist-test-slave-dfzm krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414866, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 09:27:46.763269 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:46.753624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51544 (local address 127.0.98.61:38949)
0926 09:27:46.753895 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:46.753901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:46.753936 (+ 35us) server_negotiation.cc:407] Connection header received
0926 09:27:46.753962 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:46.753965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:46.753981 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:46.754067 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:46.754691 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.755669 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.756721 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.757162 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.757227 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:46.760037 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:46.760046 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:46.760046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:46.760053 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:46.762027 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:46.762378 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:46.762382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:46.762383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:46.762457 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:46.762653 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:46.762654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:46.762655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:46.762847 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:46.763022 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:46.763070 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 09:27:46.763142 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":82,"threads_started":1}
I20260926 09:27:46.765003 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:46.757046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38949 (local address 127.0.0.1:51548)
0926 09:27:46.757086 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:46.757095 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:46.757128 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:46.757410 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:46.757410 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:46.757571 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:46.757647 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:46.757649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.758595 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.758596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:46.759226 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:46.759229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.759572 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.759573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:46.759651 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:46.760062 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:46.760066 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:46.761538 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:46.764030 (+ 2492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:46.764031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:46.764031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:46.764190 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:46.764393 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:46.764394 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:46.764394 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:46.764445 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:46.764697 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:46.764698 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:46.764735 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 09:27:46.764803 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 09:27:46.764896 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 09:27:46.765081 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:46.757126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51548 (local address 127.0.98.61:38949)
0926 09:27:46.757276 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:46.757278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:46.757285 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:27:46.757303 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:46.757304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:46.757314 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:46.757395 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:46.757715 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.758475 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.759298 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.759496 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.759552 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:46.761645 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:46.761647 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:46.761647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:46.761654 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:46.763929 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:46.764271 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:46.764272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:46.764272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:46.764322 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:46.764516 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:46.764517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:46.764518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:46.764622 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:46.764707 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:46.764828 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 09:27:46.764945 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260926 09:27:46.766331 7019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134: Bootstrap starting.
I20260926 09:27:46.766934 7019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:46.767266 7019 log.cc:824] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:46.767953 7019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134: No bootstrap required, opened a new log
I20260926 09:27:46.769413 7019 raft_consensus.cc:348] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } }
I20260926 09:27:46.769531 7019 raft_consensus.cc:374] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:46.769553 7019 raft_consensus.cc:729] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c056ade4d2f748619998a5e38bf29134, State: Initialized, Role: FOLLOWER
I20260926 09:27:46.769733 7019 consensus_queue.cc:260] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } }
I20260926 09:27:46.769980 7026 sys_catalog.cc:455] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } } }
I20260926 09:27:46.770043 7019 sys_catalog.cc:564] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:27:46.770061 7026 sys_catalog.cc:458] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 09:27:46.771538 7037 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:27:46.771593 7037 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:27:46.773232 6951 master_runner.cc:428] Detected that this master c056ade4d2f748619998a5e38bf29134 is joining an existing cluster
I20260926 09:27:46.773288 6951 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:38949
I20260926 09:27:46.781158 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:46.774611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.0.1:55650)
0926 09:27:46.774730 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:46.774740 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:46.774789 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:46.774938 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:46.774939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:46.775096 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:46.775146 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:46.775147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.775791 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.775792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:46.776348 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:46.776349 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.776734 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.776735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:46.776796 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:46.777256 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:46.777261 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:46.777658 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:46.779841 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:46.779842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:46.779843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:46.780183 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:46.780417 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:46.780419 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:46.780419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:46.780506 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:46.780810 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:46.780811 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:46.780856 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 09:27:46.780944 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 09:27:46.781042 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":81,"threads_started":1}
I20260926 09:27:46.781168 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:46.774528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.0.98.62:39437)
0926 09:27:46.774568 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:46.774570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:46.774829 (+ 259us) server_negotiation.cc:407] Connection header received
0926 09:27:46.774854 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:46.774855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:46.774867 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:46.774921 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:46.775221 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.775703 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.776442 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.776648 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.776732 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:46.777766 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:46.777768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:46.777768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:46.777775 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:46.779511 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:46.780271 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:46.780272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:46.780273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:46.780332 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:46.780600 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:46.780601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:46.780601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:46.780735 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:46.780808 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:46.780988 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 09:27:46.781073 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:27:46.781643 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:46.774289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51558 (local address 127.0.98.61:38949)
0926 09:27:46.774324 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:46.774326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:46.774486 (+ 160us) server_negotiation.cc:407] Connection header received
0926 09:27:46.774556 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:46.774556 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:46.774566 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:46.774620 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:46.775008 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.775752 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.776787 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.777413 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.777457 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:46.778824 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:46.778825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:46.778826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:46.778833 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:46.780487 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:46.780833 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:46.780834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:46.780834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:46.780876 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:46.781081 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:46.781082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:46.781082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:46.781188 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:46.781227 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:46.781416 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 09:27:46.781504 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 09:27:46.781785 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:46.774259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38949 (local address 127.0.0.1:51558)
0926 09:27:46.774418 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:46.774428 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:46.774501 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:46.774636 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:46.774637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:46.774844 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:46.774935 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:46.774937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.775851 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.775852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:46.776696 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:46.776698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.777529 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.777530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:46.777606 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:46.778131 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:46.778136 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:46.778483 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:46.780561 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:46.780562 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:46.780563 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:46.780757 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:46.780967 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:46.780968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:46.780968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:46.781017 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:46.781267 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:46.781267 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:46.781300 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 09:27:46.781395 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 09:27:46.781565 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":69,"thread_start_us":99,"threads_started":1}
Sep 26 09:27:46 dist-test-slave-dfzm krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414865, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 09:27:46.791275 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:46.783839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51568 (local address 127.0.98.61:38949)
0926 09:27:46.783880 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:46.783882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:46.784038 (+ 156us) server_negotiation.cc:407] Connection header received
0926 09:27:46.784133 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:46.784133 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:46.784141 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:46.784185 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:46.784719 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.785145 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.786304 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.786474 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.786532 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:46.788665 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:46.788666 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:46.788666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:46.788674 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:46.790080 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:46.790477 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:46.790478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:46.790478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:46.790533 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:46.790728 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:46.790729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:46.790730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:46.790834 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:46.790910 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:46.791101 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 09:27:46.791177 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:27:46.791309 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:46.783789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38949 (local address 127.0.0.1:51568)
0926 09:27:46.783942 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:46.783984 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:46.784061 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:46.784191 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:46.784194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:46.784541 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:46.784600 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:46.784602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.785226 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.785227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:46.786227 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:46.786229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.786546 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.786547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:46.786628 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:46.787062 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:46.787067 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:46.788575 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:46.790177 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:46.790184 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:46.790187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:46.790388 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:46.790599 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:46.790599 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:46.790600 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:46.790663 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:46.790903 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:46.790909 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:46.790969 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 09:27:46.791075 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 09:27:46.791199 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
I20260926 09:27:46.791932 6487 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:38949: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c056ade4d2f748619998a5e38bf29134"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 38949
}
attrs {
promote: true
}
}
dest_uuid: "705253badbc740e49445bb9ab7943235"
cas_config_opid_index: -1
I20260926 09:27:46.792285 6487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: true } }
I20260926 09:27:46.792783 7046 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c056ade4d2f748619998a5e38bf29134 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: true } } }
I20260926 09:27:46.793123 6993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:46.793889 7046 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:38949
I20260926 09:27:46.794030 7046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "705253badbc740e49445bb9ab7943235" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: true } } }
I20260926 09:27:46.794126 6993 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [term 1 FOLLOWER]: Refusing update from remote peer 705253badbc740e49445bb9ab7943235: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 09:27:46.794164 7046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:46.794189 6951 master.cc:537] Master@127.0.98.61:38949 shutting down...
I20260926 09:27:46.794584 7046 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 09:27:46.795269 6471 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 09:27:46.795806 6471 consensus_peers.cc:487] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 -> Peer c056ade4d2f748619998a5e38bf29134 (127.0.98.61:38949): Couldn't send request to peer c056ade4d2f748619998a5e38bf29134. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 09:27:46.797047 6951 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:46.797137 6951 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:46.797204 6951 tablet_replica.cc:330] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134: stopping tablet replica
I20260926 09:27:46.809832 6951 master.cc:552] Master@127.0.98.61:38949 shutdown complete.
I20260926 09:27:46.809885 6951 master_runner.cc:305] Clearing existing system tablet
I20260926 09:27:46.810030 6951 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:46.810110 6951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:46.810737 6951 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:46.811416 7051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:46.811529 6951 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:46.811587 6951 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:27:46.811619 6951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/wal
uuid: "c056ade4d2f748619998a5e38bf29134"
format_stamp: "Formatted at 2026-09-26 09:27:46 on dist-test-slave-dfzm"
server_key: "7eb333d93122e062791d6df5155e1e33"
server_key_iv: "bfc6b8fea6c2b15caa33fae0029ddf9e"
server_key_version: "encryptionkey@0"
I20260926 09:27:46.812266 6951 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:46.813700 6951 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 09:27:46.813766 6951 log.cc:1196] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 09:27:46.813972 6951 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134: Deleting consensus metadata
I20260926 09:27:46.814098 6951 master_runner.cc:315] Copying system tablet from 127.0.98.62:39437
I20260926 09:27:46.814822 6951 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:39437
I20260926 09:27:46.821409 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:46.815206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55652 (local address 127.0.98.62:39437)
0926 09:27:46.815266 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:46.815267 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:46.815316 (+ 49us) server_negotiation.cc:407] Connection header received
0926 09:27:46.815395 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:46.815396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:46.815405 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:46.815451 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:46.815814 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.816314 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.817375 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.817558 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.817686 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:46.818749 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:46.818750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:46.818750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:46.818757 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:46.820335 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:46.820743 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:46.820744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:46.820744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:46.820795 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:46.820984 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:46.820984 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:46.820984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:46.821081 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:46.821147 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:46.821286 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 09:27:46.821344 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:27:46.821485 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:46.815104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.0.1:55652)
0926 09:27:46.815251 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:46.815262 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:46.815333 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:46.815467 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:46.815468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:46.815678 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:46.815745 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:46.815747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.816393 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.816393 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:46.817285 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:46.817287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.817690 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.817691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:46.817764 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:46.818220 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:46.818225 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:46.818650 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:46.820417 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:46.820419 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:46.820419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:46.820665 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:46.820856 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:46.820857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:46.820857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:46.820924 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:46.821144 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:46.821145 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:46.821192 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 09:27:46.821265 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 09:27:46.821369 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260926 09:27:46.821990 6507 tablet_copy_service.cc:140] P 705253badbc740e49445bb9ab7943235: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c056ade4d2f748619998a5e38bf29134 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:55652)
I20260926 09:27:46.822093 6507 tablet_copy_service.cc:161] P 705253badbc740e49445bb9ab7943235: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c056ade4d2f748619998a5e38bf29134 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:55652: session id = c056ade4d2f748619998a5e38bf29134-00000000000000000000000000000000
I20260926 09:27:46.822732 6507 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 09:27:46.823390 6951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 09:27:46.824337 6951 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134: tablet copy: Starting download of 0 data blocks...
I20260926 09:27:46.824496 6951 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134: tablet copy: Starting download of 1 WAL segments...
I20260926 09:27:46.825402 6951 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 09:27:46.826285 6507 tablet_copy_service.cc:342] P 705253badbc740e49445bb9ab7943235: Request end of tablet copy session c056ade4d2f748619998a5e38bf29134-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:55652
I20260926 09:27:46.826344 6507 tablet_copy_service.cc:433] P 705253badbc740e49445bb9ab7943235: ending tablet copy session c056ade4d2f748619998a5e38bf29134-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c056ade4d2f748619998a5e38bf29134
I20260926 09:27:46.828814 6951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:46.830075 7059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:46.830152 7058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:46.830190 7061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:46.830451 6951 server_base.cc:1034] running on GCE node
I20260926 09:27:46.830571 6951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:46.830761 6951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:46.831907 6951 hybrid_clock.cc:648] HybridClock initialized: now 1790414866831893 us; error 32 us; skew 500 ppm
Sep 26 09:27:46 dist-test-slave-dfzm krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414866, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:46.833220 6951 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 09:27:46.833498 6951 webserver.cc:460] Webserver started at http://127.0.98.61:34867/ using document root <none> and password file <none>
I20260926 09:27:46.833662 6951 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:46.833703 6951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:46.834291 6951 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:46.834769 7068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:46.834888 6951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:46.834935 6951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/wal
uuid: "c056ade4d2f748619998a5e38bf29134"
format_stamp: "Formatted at 2026-09-26 09:27:46 on dist-test-slave-dfzm"
server_key: "7eb333d93122e062791d6df5155e1e33"
server_key_iv: "bfc6b8fea6c2b15caa33fae0029ddf9e"
server_key_version: "encryptionkey@0"
I20260926 09:27:46.834995 6951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:46.838630 6951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:46.840109 6951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:46.843235 6951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38949
I20260926 09:27:46.843665 7120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38949 every 8 connection(s)
I20260926 09:27:46.843705 6951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-1/data/info.pb
I20260926 09:27:46.844434 7121 sys_catalog.cc:263] Verifying existing consensus state
I20260926 09:27:46.844846 7121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134: Bootstrap starting.
I20260926 09:27:46.847054 7121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:27:46.847287 7121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134: Bootstrap complete.
I20260926 09:27:46.847481 7121 raft_consensus.cc:348] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: true } }
I20260926 09:27:46.847546 7121 raft_consensus.cc:729] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c056ade4d2f748619998a5e38bf29134, State: Initialized, Role: LEARNER
I20260926 09:27:46.847571 7121 consensus_queue.cc:260] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: true } }
I20260926 09:27:46.847764 7121 sys_catalog.cc:564] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:27:46.847745 7124 sys_catalog.cc:455] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: true } } }
I20260926 09:27:46.847821 7124 sys_catalog.cc:458] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [sys.catalog]: This master's current role is: LEARNER
I20260926 09:27:46.849241 7135 catalog_manager.cc:1259] Loaded cluster ID: 00db4655e98e4497b0dbe52a74a75983
I20260926 09:27:46.849352 7135 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134: loading cluster ID for follower catalog manager: success
I20260926 09:27:46.850358 7135 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134: acquiring CA information for follower catalog manager: success
I20260926 09:27:46.850584 7135 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 09:27:46.871933 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:46.865052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51598 (local address 127.0.98.61:38949)
0926 09:27:46.865226 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:46.865228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:46.865235 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:27:46.865258 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:46.865258 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:46.865268 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:46.865347 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:46.865983 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.866723 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.867443 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:46.867687 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:46.867770 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:46.869051 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:46.869052 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:46.869053 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:46.869060 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:46.870597 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:46.870982 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:46.870983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:46.870983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:46.871041 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:46.871250 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:46.871251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:46.871251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:46.871365 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:46.871467 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:46.871595 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 09:27:46.871683 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260926 09:27:46.898824 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39395
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39437,127.0.98.61:38949,127.0.98.60:39395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:46.990729 7139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:46.990936 7139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:46.990957 7139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:46.992129 7139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:27:46.992169 7139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:46.992184 7139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:46.992197 7139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:27:46.992208 7139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:27:46.993873 7139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39437,127.0.98.61:38949,127.0.98.60:39395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:46.994069 7139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:46.994242 7139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:46.999176 7147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:46.999182 7145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:46.999200 7144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:46.999401 7139 server_base.cc:1034] running on GCE node
I20260926 09:27:46.999632 7139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:46.999915 7139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:47.001116 7139 hybrid_clock.cc:648] HybridClock initialized: now 1790414867001088 us; error 42 us; skew 500 ppm
Sep 26 09:27:46 dist-test-slave-dfzm krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414866, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:47.003688 7139 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 09:27:47.004127 7139 webserver.cc:460] Webserver started at http://127.0.98.60:33839/ using document root <none> and password file <none>
I20260926 09:27:47.004347 7139 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:47.004412 7139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:47.004570 7139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:47.005748 7139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/data/instance:
uuid: "ad655b5c635843ce8fa735418ada7abe"
format_stamp: "Formatted at 2026-09-26 09:27:46 on dist-test-slave-dfzm"
server_key: "7d05d87e5bef22af8b86668d1aff624a"
server_key_iv: "8808a4512967aa2bac1d30aec6542339"
server_key_version: "encryptionkey@0"
I20260926 09:27:47.006139 7139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/wal/instance:
uuid: "ad655b5c635843ce8fa735418ada7abe"
format_stamp: "Formatted at 2026-09-26 09:27:46 on dist-test-slave-dfzm"
server_key: "7d05d87e5bef22af8b86668d1aff624a"
server_key_iv: "8808a4512967aa2bac1d30aec6542339"
server_key_version: "encryptionkey@0"
I20260926 09:27:47.007427 7139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:27:47.008147 7154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:47.008327 7139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:27:47.008421 7139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/wal
uuid: "ad655b5c635843ce8fa735418ada7abe"
format_stamp: "Formatted at 2026-09-26 09:27:46 on dist-test-slave-dfzm"
server_key: "7d05d87e5bef22af8b86668d1aff624a"
server_key_iv: "8808a4512967aa2bac1d30aec6542339"
server_key_version: "encryptionkey@0"
I20260926 09:27:47.008500 7139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:47.026304 7139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:47.028452 7139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:47.028627 7139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:47.033141 7139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39395
I20260926 09:27:47.033190 7206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39395 every 8 connection(s)
I20260926 09:27:47.033671 7139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/data/info.pb
I20260926 09:27:47.034070 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 7139
I20260926 09:27:47.034960 7207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:47.036924 7207 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:27:47 dist-test-slave-dfzm krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 09:27:47 dist-test-slave-dfzm krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414866, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:27:47.045711 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.037572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.0.1:55656)
0926 09:27:47.037796 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.037839 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:47.037907 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:47.038064 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:47.038066 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:47.038308 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:47.038368 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.038369 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.039122 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.039123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.039808 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.039810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.040330 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.040331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.040402 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.040870 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:47.040878 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:47.042794 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:47.044500 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.044505 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.044507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.044799 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.044987 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.044987 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.044987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.045037 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.045334 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:47.045340 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:47.045403 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 09:27:47.045483 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 09:27:47.045561 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":126,"threads_started":1}
I20260926 09:27:47.045691 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.037750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55656 (local address 127.0.98.62:39437)
0926 09:27:47.037800 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.037802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.037910 (+ 108us) server_negotiation.cc:407] Connection header received
0926 09:27:47.037971 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.037972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.037983 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.038053 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:47.038442 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.039036 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.039889 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.040229 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.040315 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.042876 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:47.042877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:47.042877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:47.042886 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:47.044427 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.044867 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.044868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.044868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.044920 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.045122 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.045123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.045123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.045257 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:47.045324 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.045515 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.045611 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:27:47.046103 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.034778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43276 (local address 127.0.98.60:39395)
0926 09:27:47.034979 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.034989 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.035021 (+ 32us) server_negotiation.cc:407] Connection header received
0926 09:27:47.035601 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.035609 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.035672 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.035864 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:47.036567 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.037895 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.038769 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.039216 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.039320 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.042228 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:47.042233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:47.042234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:47.042239 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:47.044070 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.044405 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.044408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.044409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.044512 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.044724 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.044725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.044725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.045012 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:47.045155 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.045269 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.045400 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":104,"threads_started":1}
I20260926 09:27:47.046473 7207 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:27:47 dist-test-slave-dfzm krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414866, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 09:27:47.052968 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.046719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38949 (local address 127.0.0.1:51614)
0926 09:27:47.046750 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.046759 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:47.046791 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:47.046957 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:47.046957 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:47.047089 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:47.047130 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.047131 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.047799 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.047800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.048296 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.048297 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.048675 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.048679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.048735 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.049064 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:47.049068 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:47.050340 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:47.052144 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.052145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.052145 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.052306 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.052500 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.052500 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.052501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.052557 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.052753 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:47.052753 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:47.052786 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 09:27:47.052835 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 09:27:47.052904 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260926 09:27:47.052995 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.046785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51614 (local address 127.0.98.61:38949)
0926 09:27:47.046832 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.046835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.046843 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:27:47.046868 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.046869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.046880 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.046956 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:47.047226 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.047710 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.048379 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.048592 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.048672 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.050434 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:47.050436 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:47.050436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:47.050444 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:47.052054 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.052378 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.052378 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.052378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.052428 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.052605 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.052605 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.052605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.052699 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:47.052750 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.052853 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.052924 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:27:47.053624 7207 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39395 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:27:47 dist-test-slave-dfzm krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414866, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 09:27:47.059880 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.053921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39395 (local address 127.0.0.1:43284)
0926 09:27:47.053975 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.053986 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:47.054034 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:47.054167 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:47.054167 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:47.054273 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:47.054301 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.054301 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.054917 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.054918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.055478 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.055479 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.055832 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.055833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.055868 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.056226 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:47.056230 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:47.057470 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:47.058924 (+ 1454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.058925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.058925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.059079 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.059307 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.059307 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.059308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.059360 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.059634 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:47.059634 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:47.059666 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 09:27:47.059715 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 09:27:47.059795 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 09:27:47.059990 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.054013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.0.98.60:39395)
0926 09:27:47.054066 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.054067 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.054076 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:27:47.054096 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.054096 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.054106 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.054163 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:47.054379 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.054843 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.055563 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.055763 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.055828 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.057548 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:47.057550 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:47.057550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:47.057556 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:47.058856 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.059206 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.059207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.059207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.059253 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.059435 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.059436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.059436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.059578 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:47.059630 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.059734 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.059837 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":33}
I20260926 09:27:47.061362 7207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe: Bootstrap starting.
I20260926 09:27:47.062242 7207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:47.062510 7207 log.cc:824] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:47.063194 7207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe: No bootstrap required, opened a new log
I20260926 09:27:47.064606 7207 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39395 } }
I20260926 09:27:47.064778 7207 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:47.064849 7207 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad655b5c635843ce8fa735418ada7abe, State: Initialized, Role: FOLLOWER
I20260926 09:27:47.064999 7207 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39395 } }
I20260926 09:27:47.065322 7207 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:27:47.065311 7212 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39395 } } }
I20260926 09:27:47.065532 7212 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [sys.catalog]: This master's current role is: FOLLOWER
W20260926 09:27:47.067052 7223 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:27:47.067137 7223 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:27:47.069396 7139 master_runner.cc:428] Detected that this master ad655b5c635843ce8fa735418ada7abe is joining an existing cluster
I20260926 09:27:47.069447 7139 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:39395
I20260926 09:27:47.077998 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.070541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38949 (local address 127.0.0.1:51626)
0926 09:27:47.070734 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.070747 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:47.070826 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:47.071007 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:47.071008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:47.071282 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:47.071374 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.071376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.072350 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.072350 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.073369 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.073371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.074462 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.074463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.074537 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.074953 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:47.074958 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:47.075301 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:47.076996 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.076997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.076997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.077166 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.077382 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.077383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.077383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.077447 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.077752 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:47.077752 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:47.077788 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 09:27:47.077839 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 09:27:47.077918 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":82,"thread_start_us":84,"threads_started":1}
I20260926 09:27:47.078034 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.070635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51626 (local address 127.0.98.61:38949)
0926 09:27:47.070760 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.070764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.070801 (+ 37us) server_negotiation.cc:407] Connection header received
0926 09:27:47.070924 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.070925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.070936 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.071003 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:47.071465 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.072263 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.073456 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.073845 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.073883 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.075388 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:47.075390 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:47.075390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:47.075397 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:47.076891 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.077240 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.077241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.077241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.077314 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.077510 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.077511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.077512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.077683 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:47.077762 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.077863 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.077940 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 09:27:47.097668 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 6549
I20260926 09:27:47.101761 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.070704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55660 (local address 127.0.98.62:39437)
0926 09:27:47.070785 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.070787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.071067 (+ 280us) server_negotiation.cc:407] Connection header received
0926 09:27:47.071115 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.071116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.071127 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.071177 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:47.071634 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.072372 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.073477 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.073795 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.073837 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.098058 (+ 24221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:47.098060 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:47.098060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:47.098069 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:47.099955 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.100821 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.100823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.100824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.100882 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.101093 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.101095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.101095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.101265 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:47.101353 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.101542 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.101658 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 09:27:47.102068 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.070541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.0.1:55660)
0926 09:27:47.071008 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.071017 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:47.071069 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:47.071222 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:47.071223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:47.071468 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:47.071532 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.071534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.072472 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.072473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.073377 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.073379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.073913 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.073914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.073976 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.074479 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:47.074484 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:47.097934 (+ 23450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:47.100067 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.100068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.100069 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.100743 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.100951 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.100952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.100952 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.101013 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.101382 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:47.101384 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:47.101431 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 09:27:47.101846 (+ 415us) client_negotiation.cc:240] Negotiation successful
0926 09:27:47.101959 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":4,"thread_start_us":81,"threads_started":1}
I20260926 09:27:47.104533 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.070891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39395 (local address 127.0.0.1:43294)
0926 09:27:47.071146 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.071153 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:47.071232 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:47.071386 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:47.071387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:47.071557 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:47.071608 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.071610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.072449 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.072450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.073360 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.073362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.073852 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.073853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.074355 (+ 502us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.099073 (+ 24718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:47.099078 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:47.100507 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:47.102999 (+ 2492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.102999 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.103000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.103208 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.103667 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.103667 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.103668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.103875 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.104172 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:47.104173 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:47.104208 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 09:27:47.104272 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 09:27:47.104388 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":41,"thread_start_us":67,"threads_started":1}
I20260926 09:27:47.104650 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.070910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43294 (local address 127.0.98.60:39395)
0926 09:27:47.070947 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.070949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.071213 (+ 264us) server_negotiation.cc:407] Connection header received
0926 09:27:47.071294 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.071295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.071302 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.071328 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:47.071682 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.072362 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.073452 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.073733 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.073772 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.101102 (+ 27330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:47.101103 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:47.101103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:47.101109 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:47.102916 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.103273 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.103273 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.103274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.103594 (+ 320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.103944 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.103945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.103945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.104092 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:47.104133 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.104300 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.104541 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":160,"server-negotiator.queue_time_us":21}
I20260926 09:27:47.105213 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44919
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39437
--webserver_port=40889
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:39437,127.0.98.61:38949
--tserver_master_addrs=127.0.98.62:39437,127.0.98.61:38949,127.0.98.60:39395 with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:27:47 dist-test-slave-dfzm krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414865, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 09:27:47.112151 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.104331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39395 (local address 127.0.0.1:43304)
0926 09:27:47.104375 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.104389 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:47.104437 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:47.104878 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:47.104879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:47.105031 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:47.105075 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.105076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.105941 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.105942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.106694 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.106696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.107067 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.107068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.107133 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.107528 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:47.107533 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:47.109152 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:47.111007 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.111008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.111008 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.111212 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.111440 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.111441 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.111441 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.111503 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.111828 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:47.111829 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:47.111876 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 09:27:47.111952 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 09:27:47.112052 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 09:27:47.112366 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.104424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.0.98.60:39395)
0926 09:27:47.104730 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.104732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.104739 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:27:47.104771 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.104771 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.104780 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.104831 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:47.105151 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.105874 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.106790 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.106974 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.107065 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.109247 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:47.109249 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:47.109250 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:47.109256 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:47.110914 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.111288 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.111289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.111290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.111355 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.111574 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.111575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.111576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.111752 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:47.111828 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.112042 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.112127 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":78,"threads_started":1}
I20260926 09:27:47.112824 6487 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:39395: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ad655b5c635843ce8fa735418ada7abe"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 39395
}
attrs {
promote: true
}
}
dest_uuid: "705253badbc740e49445bb9ab7943235"
cas_config_opid_index: 5
I20260926 09:27:47.113111 6487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: true } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: true } }
I20260926 09:27:47.113484 7047 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ad655b5c635843ce8fa735418ada7abe (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: true } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: true } } }
I20260926 09:27:47.113813 7181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:47.114653 7047 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:39395
I20260926 09:27:47.114748 7181 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [term 1 FOLLOWER]: Refusing update from remote peer 705253badbc740e49445bb9ab7943235: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 09:27:47.114733 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "705253badbc740e49445bb9ab7943235" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: true } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: true } } }
I20260926 09:27:47.114816 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:47.115284 7139 master.cc:537] Master@127.0.98.60:39395 shutting down...
I20260926 09:27:47.115275 7046 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 09:27:47.117489 6474 consensus_peers.cc:487] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 -> Peer ad655b5c635843ce8fa735418ada7abe (127.0.98.60:39395): Couldn't send request to peer ad655b5c635843ce8fa735418ada7abe. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 09:27:47.119643 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.115207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51630 (local address 127.0.98.61:38949)
0926 09:27:47.115250 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.115252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.115720 (+ 468us) server_negotiation.cc:407] Connection header received
0926 09:27:47.115834 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.115835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.115861 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.115905 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:47.116139 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.116946 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.118391 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.119112 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.119179 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.119250 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.119497 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.119544 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:27:47.119907 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.115019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38949 (local address 127.0.0.1:51630)
0926 09:27:47.115658 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.115670 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:47.115751 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:47.116002 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:47.116003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:47.116005 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:47.116051 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.116053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.117055 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.117056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.118290 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.118292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.119328 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.119329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.119404 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.119423 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 09:27:47.119754 (+ 331us) client_negotiation.cc:240] Negotiation successful
0926 09:27:47.119803 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625}
I20260926 09:27:47.120142 7095 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [term 1 LEARNER]: Refusing update from remote peer 705253badbc740e49445bb9ab7943235: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 09:27:47.120317 7046 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 09:27:47.120829 7095 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ad655b5c635843ce8fa735418ada7abe (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: true } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: true } } }
I20260926 09:27:47.121742 7139 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:47.121835 7139 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:47.121881 7139 tablet_replica.cc:330] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe: stopping tablet replica
I20260926 09:27:47.122253 7124 sys_catalog.cc:455] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 705253badbc740e49445bb9ab7943235. Latest consensus state: current_term: 1 leader_uuid: "705253badbc740e49445bb9ab7943235" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: true } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: true } } }
I20260926 09:27:47.122334 7124 sys_catalog.cc:458] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [sys.catalog]: This master's current role is: LEARNER
I20260926 09:27:47.122371 7124 sys_catalog.cc:455] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "705253badbc740e49445bb9ab7943235" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: true } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: true } } }
I20260926 09:27:47.122411 7124 sys_catalog.cc:458] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [sys.catalog]: This master's current role is: LEARNER
I20260926 09:27:47.123113 7046 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235: attempting to promote NON_VOTER c056ade4d2f748619998a5e38bf29134 to VOTER
I20260926 09:27:47.123302 7046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: true } }
I20260926 09:27:47.123757 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.123656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39395 (local address 127.0.0.1:43312)
0926 09:27:47.123695 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.123728 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 09:27:47.123818 7095 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [term 1 LEARNER]: Refusing update from remote peer 705253badbc740e49445bb9ab7943235: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 09:27:47.123970 7047 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 09:27:47.124290 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.123943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39395 (local address 127.0.0.1:43318)
0926 09:27:47.123969 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.123988 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260926 09:27:47.124351 6474 consensus_peers.cc:487] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 -> Peer ad655b5c635843ce8fa735418ada7abe (127.0.98.60:39395): Couldn't send request to peer ad655b5c635843ce8fa735418ada7abe. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39395: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 09:27:47.124670 7047 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c056ade4d2f748619998a5e38bf29134 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: true } } }
I20260926 09:27:47.125609 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "705253badbc740e49445bb9ab7943235" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: true } } }
I20260926 09:27:47.125687 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:47.125721 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "705253badbc740e49445bb9ab7943235" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: true } } }
I20260926 09:27:47.125764 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:47.126145 7095 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c056ade4d2f748619998a5e38bf29134 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: true } } }
I20260926 09:27:47.127045 7124 sys_catalog.cc:455] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "705253badbc740e49445bb9ab7943235" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: true } } }
I20260926 09:27:47.127122 7124 sys_catalog.cc:458] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:27:47.135839 7139 master.cc:552] Master@127.0.98.60:39395 shutdown complete.
I20260926 09:27:47.135896 7139 master_runner.cc:305] Clearing existing system tablet
I20260926 09:27:47.136021 7139 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:47.136065 7139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:47.136788 7139 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:47.137311 7238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:47.137461 7139 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:47.137509 7139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:47.137531 7139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/wal
uuid: "ad655b5c635843ce8fa735418ada7abe"
format_stamp: "Formatted at 2026-09-26 09:27:46 on dist-test-slave-dfzm"
server_key: "7d05d87e5bef22af8b86668d1aff624a"
server_key_iv: "8808a4512967aa2bac1d30aec6542339"
server_key_version: "encryptionkey@0"
I20260926 09:27:47.138239 7139 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:47.139719 7139 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 09:27:47.139786 7139 log.cc:1196] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 09:27:47.140038 7139 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe: Deleting consensus metadata
I20260926 09:27:47.140152 7139 master_runner.cc:315] Copying system tablet from 127.0.98.62:39437
I20260926 09:27:47.140733 7139 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:39437
I20260926 09:27:47.148720 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.141025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.0.1:55668)
0926 09:27:47.141714 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.141727 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:47.141803 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:47.141947 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:47.141948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:47.142158 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:47.142208 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.142210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.143171 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.143172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.144064 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.144066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.144535 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.144536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.144593 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.145146 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:47.145151 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:47.145548 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:47.147455 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.147456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.147456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.147674 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.147941 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.147942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.147943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.148017 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.148384 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:47.148385 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:47.148430 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 09:27:47.148498 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 09:27:47.148601 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":72,"threads_started":1}
I20260926 09:27:47.148725 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.141116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55668 (local address 127.0.98.62:39437)
0926 09:27:47.141171 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.141174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.141791 (+ 617us) server_negotiation.cc:407] Connection header received
0926 09:27:47.141859 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.141861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.141873 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.141934 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:47.142294 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.143072 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.144158 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.144449 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.144529 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.145681 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:47.145682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:47.145682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:47.145690 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:47.147367 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.147802 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.147803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.147803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.147866 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.148090 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.148091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.148091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.148224 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:47.148336 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.148547 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.148628 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:27:47.149395 6507 tablet_copy_service.cc:140] P 705253badbc740e49445bb9ab7943235: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ad655b5c635843ce8fa735418ada7abe ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:55668)
I20260926 09:27:47.149472 6507 tablet_copy_service.cc:161] P 705253badbc740e49445bb9ab7943235: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ad655b5c635843ce8fa735418ada7abe at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:55668: session id = ad655b5c635843ce8fa735418ada7abe-00000000000000000000000000000000
I20260926 09:27:47.150064 6507 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 09:27:47.150503 7139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 09:27:47.151790 7139 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe: tablet copy: Starting download of 0 data blocks...
I20260926 09:27:47.151930 7139 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe: tablet copy: Starting download of 1 WAL segments...
I20260926 09:27:47.152659 7139 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 09:27:47.153460 6507 tablet_copy_service.cc:342] P 705253badbc740e49445bb9ab7943235: Request end of tablet copy session ad655b5c635843ce8fa735418ada7abe-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:55668
I20260926 09:27:47.153517 6507 tablet_copy_service.cc:433] P 705253badbc740e49445bb9ab7943235: ending tablet copy session ad655b5c635843ce8fa735418ada7abe-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ad655b5c635843ce8fa735418ada7abe
I20260926 09:27:47.157007 7139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:47.158190 7248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:47.158234 7245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:47.158699 7246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:47.158804 7139 server_base.cc:1034] running on GCE node
I20260926 09:27:47.158967 7139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:47.159116 7139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:47.160246 7139 hybrid_clock.cc:648] HybridClock initialized: now 1790414867160237 us; error 27 us; skew 500 ppm
Sep 26 09:27:47 dist-test-slave-dfzm krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414867, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:47.161866 7139 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 09:27:47.162151 7139 webserver.cc:460] Webserver started at http://127.0.98.60:36277/ using document root <none> and password file <none>
I20260926 09:27:47.162276 7139 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:47.162315 7139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:47.163091 7139 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:47.163674 7255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:47.163842 7139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:47.163898 7139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/wal
uuid: "ad655b5c635843ce8fa735418ada7abe"
format_stamp: "Formatted at 2026-09-26 09:27:46 on dist-test-slave-dfzm"
server_key: "7d05d87e5bef22af8b86668d1aff624a"
server_key_iv: "8808a4512967aa2bac1d30aec6542339"
server_key_version: "encryptionkey@0"
I20260926 09:27:47.163957 7139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:47.192821 7139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:47.194793 7139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:47.197994 7139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39395
I20260926 09:27:47.198101 7307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39395 every 8 connection(s)
I20260926 09:27:47.198330 7139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/master-2/data/info.pb
I20260926 09:27:47.199239 7308 sys_catalog.cc:263] Verifying existing consensus state
I20260926 09:27:47.199759 7308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe: Bootstrap starting.
I20260926 09:27:47.202814 7308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:27:47.203022 7308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe: Bootstrap complete.
I20260926 09:27:47.203227 7308 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: true } }
I20260926 09:27:47.203320 7308 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ad655b5c635843ce8fa735418ada7abe, State: Initialized, Role: LEARNER
I20260926 09:27:47.203357 7308 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: true } }
I20260926 09:27:47.203601 7308 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:27:47.205245 7322 catalog_manager.cc:1259] Loaded cluster ID: 00db4655e98e4497b0dbe52a74a75983
I20260926 09:27:47.205298 7322 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe: loading cluster ID for follower catalog manager: success
I20260926 09:27:47.206491 7322 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe: acquiring CA information for follower catalog manager: success
I20260926 09:27:47.206806 7322 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 09:27:47.206965 7311 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: true } } }
I20260926 09:27:47.207060 7311 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [sys.catalog]: This master's current role is: LEARNER
W20260926 09:27:47.209038 7233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:47.209257 7233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:47.209309 7233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:47.210577 7233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:47.210640 7233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:47.210688 7233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:27:47.212194 7233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40889
--tserver_master_addrs=127.0.98.62:39437,127.0.98.61:38949,127.0.98.60:39395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:47.212460 7233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:47.212702 7233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:47.217805 7330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:47.217860 7233 server_base.cc:1034] running on GCE node
W20260926 09:27:47.217792 7328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:47.217821 7327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:47.218185 7233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:47.218477 7233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:47.219760 7233 hybrid_clock.cc:648] HybridClock initialized: now 1790414867219742 us; error 40 us; skew 500 ppm
Sep 26 09:27:47 dist-test-slave-dfzm krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414867, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:47.222441 7233 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 09:27:47.222889 7233 webserver.cc:460] Webserver started at http://127.0.98.1:40889/ using document root <none> and password file <none>
I20260926 09:27:47.223124 7233 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:47.223217 7233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:47.224697 7233 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:27:47.225381 7337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:47.225569 7233 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:27:47.225718 7233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "d7110ef2e34945f9958fbbcfec4c6f4d"
format_stamp: "Formatted at 2026-09-26 09:27:46 on dist-test-slave-dfzm"
server_key: "408508d91002cf4e43590d285e00ead4"
server_key_iv: "7cf08a35f4c046c3e3449f2f61403098"
server_key_version: "encryptionkey@0"
I20260926 09:27:47.226109 7233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:47.248545 7233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:47.250808 7233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:47.250977 7233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:47.251241 7233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:47.251564 7233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:47.251619 7233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:47.251664 7233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:47.251688 7233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:47.257791 7233 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44919
I20260926 09:27:47.257835 7450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44919 every 8 connection(s)
I20260926 09:27:47.258198 7233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:27:47.264361 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 7233
I20260926 09:27:47.264536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 6683
Sep 26 09:27:47 dist-test-slave-dfzm krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414867, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 09:27:47 dist-test-slave-dfzm krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414867, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 09:27:47 dist-test-slave-dfzm krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414867, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 09:27:47.270859 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.259201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51867 (local address 127.0.98.62:39437)
0926 09:27:47.259257 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.259259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.259391 (+ 132us) server_negotiation.cc:407] Connection header received
0926 09:27:47.260007 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.260009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.260022 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.260059 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:47.260849 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.261344 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.262751 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.263043 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.263103 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.266181 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:47.266183 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:47.266183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:47.266192 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:47.268614 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.269765 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.269766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.269767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.269842 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.270093 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.270094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.270094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.270273 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:47.270330 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.270674 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.270761 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 09:27:47.270859 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.259040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40915 (local address 127.0.98.61:38949)
0926 09:27:47.259141 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.259143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.259379 (+ 236us) server_negotiation.cc:407] Connection header received
0926 09:27:47.259992 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.259994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.260005 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.260058 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:47.260862 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.261629 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.263156 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.263340 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.263461 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.267400 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:47.267401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:47.267402 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:47.267410 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:47.269108 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.269767 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.269768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.269768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.269842 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.270091 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.270092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.270092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.270235 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:47.270283 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.270682 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.270761 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 09:27:47.271095 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.258965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39395 (local address 127.0.98.1:57311)
0926 09:27:47.259530 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.259538 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:47.259903 (+ 365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:47.260556 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:47.260557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:47.260696 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:47.260750 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.260755 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.261917 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.261918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.262530 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.262532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.263024 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.263025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.263158 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.263907 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:47.263917 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:47.266338 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:47.268340 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.268347 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.268349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.269650 (+ 1301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.269865 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.269866 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.269866 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.270010 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.270330 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:47.270336 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:47.270513 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 09:27:47.270625 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 09:27:47.270814 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":221,"thread_start_us":73,"threads_started":1}
I20260926 09:27:47.271231 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.258965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.98.1:51867)
0926 09:27:47.259315 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.259328 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:47.259901 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:47.260156 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:47.260157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:47.260540 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:47.260749 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.260753 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.261453 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.261454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.262682 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.262683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.263114 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.263114 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.263168 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.263907 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:47.263917 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:47.266065 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:47.268697 (+ 2632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.268698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.268699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.269664 (+ 965us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.269919 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.269919 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.269919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.270012 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.270361 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:47.270362 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:47.270511 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 09:27:47.270940 (+ 429us) client_negotiation.cc:240] Negotiation successful
0926 09:27:47.271066 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":35,"thread_start_us":86,"threads_started":1}
I20260926 09:27:47.271231 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.258965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38949 (local address 127.0.98.1:40915)
0926 09:27:47.259237 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.259279 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:47.259880 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:47.260135 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:47.260139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:47.260542 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:47.260749 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.260755 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.261702 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.261702 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.262666 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.262668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.263876 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.263877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.263937 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.265703 (+ 1766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:47.265708 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:47.267273 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:47.269233 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.269234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.269234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.269679 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.269913 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.269914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.269915 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.270012 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.270335 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:47.270336 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:47.270511 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 09:27:47.271010 (+ 499us) client_negotiation.cc:240] Negotiation successful
0926 09:27:47.271118 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":67,"thread_start_us":95,"threads_started":1}
I20260926 09:27:47.271598 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.259016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57311 (local address 127.0.98.60:39395)
0926 09:27:47.259483 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.259485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.259609 (+ 124us) server_negotiation.cc:407] Connection header received
0926 09:27:47.260046 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.260047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.260059 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.260129 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:47.260862 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.261839 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.262636 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.262944 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.263011 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.266436 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:47.266438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:47.266438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:47.266445 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:47.268241 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.269734 (+ 1493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.269736 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.269736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.269800 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.270083 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.270083 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.270084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.270246 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:47.270305 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.271353 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.271454 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":75,"threads_started":1}
I20260926 09:27:47.271628 7453 heartbeater.cc:347] Connected to a master server at 127.0.98.61:38949
I20260926 09:27:47.271723 7453 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:47.271912 7453 heartbeater.cc:511] Master 127.0.98.61:38949 requested a full tablet report, sending...
I20260926 09:27:47.272176 7451 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39395
I20260926 09:27:47.272207 7451 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:47.272248 7451 heartbeater.cc:511] Master 127.0.98.60:39395 requested a full tablet report, sending...
I20260926 09:27:47.272639 7085 ts_manager.cc:194] Registered new tserver with Master: d7110ef2e34945f9958fbbcfec4c6f4d (127.0.98.1:44919)
I20260926 09:27:47.272686 7272 ts_manager.cc:194] Registered new tserver with Master: d7110ef2e34945f9958fbbcfec4c6f4d (127.0.98.1:44919)
I20260926 09:27:47.273192 7452 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39437
I20260926 09:27:47.273242 7452 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:47.273314 7452 heartbeater.cc:511] Master 127.0.98.62:39437 requested a full tablet report, sending...
I20260926 09:27:47.273662 6487 ts_manager.cc:194] Re-registered known tserver with Master: d7110ef2e34945f9958fbbcfec4c6f4d (127.0.98.1:44919)
I20260926 09:27:47.273757 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45101
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39437
--webserver_port=37627
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:39437,127.0.98.61:38949
--tserver_master_addrs=127.0.98.62:39437,127.0.98.61:38949,127.0.98.60:39395 with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:27:47.274132 6487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51867
W20260926 09:27:47.355207 7459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:47.355418 7459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:47.355443 7459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:47.356639 7459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:47.356686 7459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:47.356748 7459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:27:47.358280 7459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37627
--tserver_master_addrs=127.0.98.62:39437,127.0.98.61:38949,127.0.98.60:39395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:47.358525 7459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:47.358801 7459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:47.364100 7465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:47.364108 7467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:47.364172 7464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:47.364319 7459 server_base.cc:1034] running on GCE node
I20260926 09:27:47.364519 7459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:47.364779 7459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:47.365940 7459 hybrid_clock.cc:648] HybridClock initialized: now 1790414867365938 us; error 48 us; skew 500 ppm
Sep 26 09:27:47 dist-test-slave-dfzm krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414867, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:47.368480 7459 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 09:27:47.368934 7459 webserver.cc:460] Webserver started at http://127.0.98.2:37627/ using document root <none> and password file <none>
I20260926 09:27:47.369117 7459 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:47.369167 7459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:47.370525 7459 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:47.371074 7474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:47.371256 7459 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:47.371347 7459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "4094f93fa463479dbaed35e8f6e5a437"
format_stamp: "Formatted at 2026-09-26 09:27:46 on dist-test-slave-dfzm"
server_key: "fde31024897de9e983a8667d1b1c9886"
server_key_iv: "4174bbf8cf7e217224c8f630b2b23240"
server_key_version: "encryptionkey@0"
I20260926 09:27:47.371693 7459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:47.378960 7459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:47.380728 7459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:47.380872 7459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:47.381112 7459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:47.381413 7459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:47.381469 7459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:47.381510 7459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:47.381538 7459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:47.387120 7459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45101
I20260926 09:27:47.387190 7587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45101 every 8 connection(s)
I20260926 09:27:47.387542 7459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:27:47.388224 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 7459
I20260926 09:27:47.388391 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 6817
I20260926 09:27:47.394472 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36285
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39437
--webserver_port=33597
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:39437,127.0.98.61:38949
--tserver_master_addrs=127.0.98.62:39437,127.0.98.61:38949,127.0.98.60:39395 with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:27:47 dist-test-slave-dfzm krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414867, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 09:27:47 dist-test-slave-dfzm krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414867, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 09:27:47 dist-test-slave-dfzm krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414867, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 09:27:47.400695 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.388351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57971 (local address 127.0.98.60:39395)
0926 09:27:47.388539 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.388542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.388613 (+ 71us) server_negotiation.cc:407] Connection header received
0926 09:27:47.389396 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.389397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.389412 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.389443 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:47.390874 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.391638 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.393292 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.393541 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.393620 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.397171 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:47.397173 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:47.397173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:47.397181 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:47.398990 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.399389 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.399390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.399391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.399449 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.399758 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.399759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.399759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.399904 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:47.399949 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.400475 (+ 526us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.400567 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260926 09:27:47.401505 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.388295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39395 (local address 127.0.98.2:57971)
0926 09:27:47.388548 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.388559 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:47.389291 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:47.389617 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:47.389622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:47.390478 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:47.390737 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.390740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.391755 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.391756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.393180 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.393183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.393693 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.393694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.393848 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.395243 (+ 1395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:47.395253 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:47.397060 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:47.399064 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.399072 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.399074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.399297 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.399531 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.399532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.399532 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.399667 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.400003 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:47.400009 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:47.400244 (+ 235us) client_negotiation.cc:769] Sending connection context
0926 09:27:47.401007 (+ 763us) client_negotiation.cc:240] Negotiation successful
0926 09:27:47.401219 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":5,"spinlock_wait_cycles":9472,"thread_start_us":96,"threads_started":1}
I20260926 09:27:47.402369 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.389350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56551 (local address 127.0.98.62:39437)
0926 09:27:47.389943 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.389945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.389955 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:27:47.389979 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.389980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.389991 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.390049 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:47.390833 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.391638 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.394356 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.394695 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.394745 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.398585 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:47.398586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:47.398587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:47.398595 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:47.400401 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.400835 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.400836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.400837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.400899 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.401356 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.401357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.401357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.401487 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:47.401532 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.402195 (+ 663us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.402284 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566}
I20260926 09:27:47.402531 7588 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39395
I20260926 09:27:47.402623 7588 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:47.402814 7588 heartbeater.cc:511] Master 127.0.98.60:39395 requested a full tablet report, sending...
I20260926 09:27:47.403208 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.388295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.98.2:56551)
0926 09:27:47.388473 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.388523 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:47.389272 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:47.390434 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:47.390436 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:47.390592 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:47.390734 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.390739 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.393283 (+ 2544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.393283 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.394250 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.394253 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.394824 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.394825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.394889 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.396903 (+ 2014us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:47.396908 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:47.398475 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:47.400482 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.400482 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.400483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.400741 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.400965 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.400965 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.400966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.401017 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.401610 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:47.401611 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:47.401664 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 09:27:47.403010 (+ 1346us) client_negotiation.cc:240] Negotiation successful
0926 09:27:47.403116 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260926 09:27:47.403290 7272 ts_manager.cc:194] Registered new tserver with Master: 4094f93fa463479dbaed35e8f6e5a437 (127.0.98.2:45101)
I20260926 09:27:47.403513 7589 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39437
I20260926 09:27:47.403558 7589 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:47.403630 7589 heartbeater.cc:511] Master 127.0.98.62:39437 requested a full tablet report, sending...
I20260926 09:27:47.403833 6487 ts_manager.cc:194] Re-registered known tserver with Master: 4094f93fa463479dbaed35e8f6e5a437 (127.0.98.2:45101)
I20260926 09:27:47.404070 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.388935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42141 (local address 127.0.98.61:38949)
0926 09:27:47.388981 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.388983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.394310 (+ 5327us) server_negotiation.cc:407] Connection header received
0926 09:27:47.394444 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.394446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.394461 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.394528 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:47.395146 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.395862 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.396851 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.397192 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.397262 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.400819 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:47.400821 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:47.400822 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:47.400830 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:47.402646 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.403080 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.403081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.403082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.403140 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.403415 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.403417 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.403417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.403564 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:47.403613 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.403842 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.403925 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:27:47.404284 6487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56551
I20260926 09:27:47.404522 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.388785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38949 (local address 127.0.98.2:42141)
0926 09:27:47.394245 (+ 5460us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.394258 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:47.394338 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:47.394758 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:47.394759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:47.394955 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:47.395015 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.395017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.395953 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.395954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.396768 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.396771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.397319 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.397320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.397395 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.398350 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:47.398355 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:47.400707 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:47.402750 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.402751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.402752 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.402986 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.403234 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.403235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.403236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.403300 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.403668 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:47.403669 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:47.403716 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 09:27:47.404166 (+ 450us) client_negotiation.cc:240] Negotiation successful
0926 09:27:47.404334 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5434,"thread_start_us":64,"threads_started":1}
I20260926 09:27:47.405082 7590 heartbeater.cc:347] Connected to a master server at 127.0.98.61:38949
I20260926 09:27:47.405134 7590 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:47.405190 7590 heartbeater.cc:511] Master 127.0.98.61:38949 requested a full tablet report, sending...
I20260926 09:27:47.405483 7085 ts_manager.cc:194] Registered new tserver with Master: 4094f93fa463479dbaed35e8f6e5a437 (127.0.98.2:45101)
W20260926 09:27:47.479513 7595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:47.479756 7595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:47.479794 7595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:47.480919 7595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:47.480989 7595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:47.481031 7595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:27:47.482542 7595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36285
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33597
--tserver_master_addrs=127.0.98.62:39437,127.0.98.61:38949,127.0.98.60:39395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:47.482812 7595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:47.483053 7595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:27:47.488268 7595 server_base.cc:1034] running on GCE node
W20260926 09:27:47.488174 7601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:47.488132 7600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:47.488131 7603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:47.488637 7595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:47.488945 7595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:47.490105 7595 hybrid_clock.cc:648] HybridClock initialized: now 1790414867490099 us; error 44 us; skew 500 ppm
Sep 26 09:27:47 dist-test-slave-dfzm krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414867, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:47.492908 7595 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 09:27:47.493364 7595 webserver.cc:460] Webserver started at http://127.0.98.3:33597/ using document root <none> and password file <none>
I20260926 09:27:47.493549 7595 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:47.493645 7595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:47.495011 7595 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:47.495605 7610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:47.495772 7595 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:47.495877 7595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "a9f71727a3f743c2b04bc0b854fdbad6"
format_stamp: "Formatted at 2026-09-26 09:27:46 on dist-test-slave-dfzm"
server_key: "9aba6fd1f95f5a7608c68234d1f27435"
server_key_iv: "397a4246be307ab0fd3dea3e0f1fba58"
server_key_version: "encryptionkey@0"
I20260926 09:27:47.496250 7595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:47.516248 7595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:47.518301 7595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:47.518468 7595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:47.518711 7595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:47.519021 7595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:47.519075 7595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:47.519121 7595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:47.519146 7595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:47.525318 7595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36285
I20260926 09:27:47.525362 7723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36285 every 8 connection(s)
I20260926 09:27:47.525767 7595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:27:47.529888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 7595
Sep 26 09:27:47 dist-test-slave-dfzm krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414867, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 09:27:47 dist-test-slave-dfzm krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414867, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 09:27:47 dist-test-slave-dfzm krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414867, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:27:47.539698 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.526473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39395 (local address 127.0.98.3:33463)
0926 09:27:47.526920 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.526931 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:47.528056 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:47.528313 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:47.528316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:47.528745 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:47.528939 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.528944 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.530019 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.530020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.531167 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.531170 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.531711 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.531712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.531845 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.532582 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:47.532588 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:47.535431 (+ 2843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:47.537840 (+ 2409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.537841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.537841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.538644 (+ 803us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.538858 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.538859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.538859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.538914 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.539185 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:47.539186 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:47.539229 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 09:27:47.539392 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 09:27:47.539500 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":87,"thread_start_us":258,"threads_started":1}
I20260926 09:27:47.539765 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.527185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35933 (local address 127.0.98.61:38949)
0926 09:27:47.527229 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.527230 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.527681 (+ 451us) server_negotiation.cc:407] Connection header received
0926 09:27:47.528180 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.528181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.528195 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.528228 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:47.529088 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.529837 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.531289 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.531595 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.531638 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.535892 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:47.535894 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:47.535894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:47.535902 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:47.537561 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.538211 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.538212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.538213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.538270 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.538556 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.538558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.538558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.538702 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:47.538766 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.539569 (+ 803us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.539663 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:27:47.540038 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.526786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33463 (local address 127.0.98.60:39395)
0926 09:27:47.526834 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.526836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.526998 (+ 162us) server_negotiation.cc:407] Connection header received
0926 09:27:47.528154 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.528155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.528166 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.528206 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:47.529031 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.529870 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.531287 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.531575 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.531624 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.535521 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:47.535523 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:47.535523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:47.535530 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:47.537751 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.538719 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.538720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.538720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.538790 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.538976 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.538976 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.538977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.539106 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:47.539160 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.539835 (+ 675us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.539927 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:27:47.540292 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.527105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38949 (local address 127.0.98.3:35933)
0926 09:27:47.527225 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.527234 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:47.528063 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:47.528315 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:47.528318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:47.528771 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:47.528940 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.528945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.529970 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.529971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.531170 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.531173 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.531721 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.531722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.531846 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.532465 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:47.532475 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:47.535785 (+ 3310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:47.537681 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.537688 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.537691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.538131 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.538340 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.538341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.538341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.538470 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.538794 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:47.538801 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:47.538972 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 09:27:47.539095 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 09:27:47.539415 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":45,"thread_start_us":76,"threads_started":1}
I20260926 09:27:47.540386 7724 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39395
I20260926 09:27:47.540486 7724 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:47.540591 7726 heartbeater.cc:347] Connected to a master server at 127.0.98.61:38949
I20260926 09:27:47.540627 7726 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:47.540705 7724 heartbeater.cc:511] Master 127.0.98.60:39395 requested a full tablet report, sending...
I20260926 09:27:47.540704 7726 heartbeater.cc:511] Master 127.0.98.61:38949 requested a full tablet report, sending...
I20260926 09:27:47.541199 7272 ts_manager.cc:194] Registered new tserver with Master: a9f71727a3f743c2b04bc0b854fdbad6 (127.0.98.3:36285)
I20260926 09:27:47.541221 7085 ts_manager.cc:194] Registered new tserver with Master: a9f71727a3f743c2b04bc0b854fdbad6 (127.0.98.3:36285)
I20260926 09:27:47.541646 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.526938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50993 (local address 127.0.98.62:39437)
0926 09:27:47.526985 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.526987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.526997 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:27:47.528180 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.528181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.528195 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.528228 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:47.529051 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.529656 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.532236 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.532562 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.532606 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.535710 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:47.535711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:47.535712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:47.535719 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:47.537900 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.540600 (+ 2700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.540601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.540602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.540672 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.540950 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.540951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.540951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.541086 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:47.541128 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.541426 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.541505 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":25}
I20260926 09:27:47.542521 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.526473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.98.3:50993)
0926 09:27:47.526841 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.526883 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:47.528054 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:47.528315 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:47.528318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:47.528770 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:47.528940 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.528945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.531366 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.531367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.532140 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.532142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.532684 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.532685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.532766 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.533194 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:47.533199 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:47.535609 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:47.538005 (+ 2396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.538006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.538006 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.540529 (+ 2523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.540792 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:47.540793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:47.540794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:47.540852 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:47.541194 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:47.541195 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:47.541238 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 09:27:47.541959 (+ 721us) client_negotiation.cc:240] Negotiation successful
0926 09:27:47.542076 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":31,"thread_start_us":285,"threads_started":1}
I20260926 09:27:47.543107 7725 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39437
I20260926 09:27:47.543161 7725 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:47.543237 7725 heartbeater.cc:511] Master 127.0.98.62:39437 requested a full tablet report, sending...
I20260926 09:27:47.543502 6487 ts_manager.cc:194] Re-registered known tserver with Master: a9f71727a3f743c2b04bc0b854fdbad6 (127.0.98.3:36285)
I20260926 09:27:47.543815 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.531004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55670 (local address 127.0.98.62:39437)
0926 09:27:47.531293 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.531295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.531457 (+ 162us) server_negotiation.cc:407] Connection header received
0926 09:27:47.531476 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.531477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.531488 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.531541 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:47.532430 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.533157 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.534297 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.534648 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.534720 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.536277 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:47.536279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:47.536279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:47.536285 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:47.539751 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.540157 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.540158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.540158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.540221 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.542182 (+ 1961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.542182 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.542183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.542320 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:47.543566 (+ 1246us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.543608 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.543686 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":76,"threads_started":1}
I20260926 09:27:47.544010 6487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50993
I20260926 09:27:47.546360 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.531135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51634 (local address 127.0.98.61:38949)
0926 09:27:47.533792 (+ 2657us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.533794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.534360 (+ 566us) server_negotiation.cc:407] Connection header received
0926 09:27:47.534416 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.534417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.534429 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.534496 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:47.536017 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.536918 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.540649 (+ 3731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.540977 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.541018 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.542501 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:47.542502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:47.542502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:47.542509 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:47.544514 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.544909 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.544911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.544911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.544970 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.545244 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.545245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.545245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.545383 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:47.546133 (+ 750us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.546178 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.546263 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2640,"thread_start_us":56,"threads_started":1}
I20260926 09:27:47.546569 6486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:27:47.548310 6486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:27:47.550474 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.532254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.0.98.60:39395)
0926 09:27:47.533833 (+ 1579us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.533835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.536283 (+ 2448us) server_negotiation.cc:407] Connection header received
0926 09:27:47.536683 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.536684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.536696 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.536738 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:47.538656 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.539556 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.542318 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.542621 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.542697 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.544090 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:47.544091 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:47.544092 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:47.544098 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:47.549125 (+ 5027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.549560 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.549561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.549561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.549640 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.549853 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.549854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.549854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.550002 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:47.550101 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.550274 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.550367 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1565,"thread_start_us":75,"threads_started":1}
I20260926 09:27:47.556788 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.552910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40692 (local address 127.0.98.2:45101)
0926 09:27:47.553483 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.553487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.553494 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:27:47.553523 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.553524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.553535 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.553649 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:47.553803 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.554537 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.555874 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.556391 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.556501 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.556588 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.556663 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.556702 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":69,"threads_started":1}
I20260926 09:27:47.556823 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.552535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36285 (local address 127.0.0.1:43022)
0926 09:27:47.552568 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.552578 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:47.552615 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:47.553012 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:47.553013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:47.553015 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:47.553064 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.553065 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.554452 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.554453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.555732 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.555735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.556582 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.556583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.556649 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.556652 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:27:47.556681 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 09:27:47.556724 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":95}
I20260926 09:27:47.557005 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.552950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41130 (local address 127.0.98.1:44919)
0926 09:27:47.553115 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.553118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.553163 (+ 45us) server_negotiation.cc:407] Connection header received
0926 09:27:47.553199 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.553200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.553213 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.553281 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:47.553433 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.554519 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.555694 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.556469 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.556554 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.556627 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.556864 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.556914 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260926 09:27:47.556821 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.552820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45101 (local address 127.0.0.1:40692)
0926 09:27:47.553353 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.553362 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:47.553394 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:47.553683 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:47.553684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:47.553686 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:47.553729 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.553731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.554630 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.554631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.555813 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.555815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.556503 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.556504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.556575 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.556581 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 09:27:47.556648 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 09:27:47.556691 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":44,"thread_start_us":65,"threads_started":1}
I20260926 09:27:47.557458 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.552619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44919 (local address 127.0.0.1:41130)
0926 09:27:47.552756 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.552766 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:47.553082 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:47.553298 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:47.553299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:47.553301 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:47.553348 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.553350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.554607 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.554608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.555595 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.555597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.556656 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.556657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.556731 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.556735 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:27:47.556782 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 09:27:47.556817 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":68,"thread_start_us":96,"threads_started":1}
I20260926 09:27:47.557473 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.552663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43022 (local address 127.0.98.3:36285)
0926 09:27:47.552833 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.552839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.552849 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:27:47.552890 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.552892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.552909 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.553140 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:47.553192 (+ 52us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.554363 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.555818 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.556492 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.557129 (+ 637us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.557293 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.557343 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.557394 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":78,"threads_started":1}
I20260926 09:27:47.558094 7522 tablet_service.cc:1467] Processing CreateTablet for tablet 9efb814c6b8447ca8cfb6d2f2dc01031 (DEFAULT_TABLE table=test-table [id=b3a95e2536854cca8cf97eaadf106797]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:47.558188 7654 tablet_service.cc:1467] Processing CreateTablet for tablet 9efb814c6b8447ca8cfb6d2f2dc01031 (DEFAULT_TABLE table=test-table [id=b3a95e2536854cca8cf97eaadf106797]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:47.558480 7654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9efb814c6b8447ca8cfb6d2f2dc01031. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:47.558490 7522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9efb814c6b8447ca8cfb6d2f2dc01031. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:47.558470 7385 tablet_service.cc:1467] Processing CreateTablet for tablet 9efb814c6b8447ca8cfb6d2f2dc01031 (DEFAULT_TABLE table=test-table [id=b3a95e2536854cca8cf97eaadf106797]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:47.558786 7385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9efb814c6b8447ca8cfb6d2f2dc01031. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:47.561466 7750 tablet_bootstrap.cc:492] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6: Bootstrap starting.
I20260926 09:27:47.562052 7750 tablet_bootstrap.cc:654] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:47.562378 7750 log.cc:824] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:47.562521 7751 tablet_bootstrap.cc:492] T 9efb814c6b8447ca8cfb6d2f2dc01031 P 4094f93fa463479dbaed35e8f6e5a437: Bootstrap starting.
I20260926 09:27:47.562692 7752 tablet_bootstrap.cc:492] T 9efb814c6b8447ca8cfb6d2f2dc01031 P d7110ef2e34945f9958fbbcfec4c6f4d: Bootstrap starting.
I20260926 09:27:47.563141 7751 tablet_bootstrap.cc:654] T 9efb814c6b8447ca8cfb6d2f2dc01031 P 4094f93fa463479dbaed35e8f6e5a437: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:47.563268 7752 tablet_bootstrap.cc:654] T 9efb814c6b8447ca8cfb6d2f2dc01031 P d7110ef2e34945f9958fbbcfec4c6f4d: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:47.563516 7751 log.cc:824] T 9efb814c6b8447ca8cfb6d2f2dc01031 P 4094f93fa463479dbaed35e8f6e5a437: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:47.563576 7752 log.cc:824] T 9efb814c6b8447ca8cfb6d2f2dc01031 P d7110ef2e34945f9958fbbcfec4c6f4d: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:47.564309 7751 tablet_bootstrap.cc:492] T 9efb814c6b8447ca8cfb6d2f2dc01031 P 4094f93fa463479dbaed35e8f6e5a437: No bootstrap required, opened a new log
I20260926 09:27:47.564309 7750 tablet_bootstrap.cc:492] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6: No bootstrap required, opened a new log
I20260926 09:27:47.564381 7750 ts_tablet_manager.cc:1397] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 09:27:47.564381 7751 ts_tablet_manager.cc:1397] T 9efb814c6b8447ca8cfb6d2f2dc01031 P 4094f93fa463479dbaed35e8f6e5a437: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:27:47.565904 7752 tablet_bootstrap.cc:492] T 9efb814c6b8447ca8cfb6d2f2dc01031 P d7110ef2e34945f9958fbbcfec4c6f4d: No bootstrap required, opened a new log
I20260926 09:27:47.565989 7752 ts_tablet_manager.cc:1397] T 9efb814c6b8447ca8cfb6d2f2dc01031 P d7110ef2e34945f9958fbbcfec4c6f4d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 09:27:47.566452 7750 raft_consensus.cc:348] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44919 } } peers { permanent_uuid: "4094f93fa463479dbaed35e8f6e5a437" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45101 } }
I20260926 09:27:47.566622 7750 raft_consensus.cc:374] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:47.566658 7750 raft_consensus.cc:729] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f71727a3f743c2b04bc0b854fdbad6, State: Initialized, Role: FOLLOWER
I20260926 09:27:47.566798 7750 consensus_queue.cc:260] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44919 } } peers { permanent_uuid: "4094f93fa463479dbaed35e8f6e5a437" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45101 } }
I20260926 09:27:47.567050 7750 ts_tablet_manager.cc:1428] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:27:47.566953 7751 raft_consensus.cc:348] T 9efb814c6b8447ca8cfb6d2f2dc01031 P 4094f93fa463479dbaed35e8f6e5a437 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44919 } } peers { permanent_uuid: "4094f93fa463479dbaed35e8f6e5a437" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45101 } }
I20260926 09:27:47.567098 7751 raft_consensus.cc:374] T 9efb814c6b8447ca8cfb6d2f2dc01031 P 4094f93fa463479dbaed35e8f6e5a437 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:47.567130 7751 raft_consensus.cc:729] T 9efb814c6b8447ca8cfb6d2f2dc01031 P 4094f93fa463479dbaed35e8f6e5a437 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4094f93fa463479dbaed35e8f6e5a437, State: Initialized, Role: FOLLOWER
I20260926 09:27:47.567250 7751 consensus_queue.cc:260] T 9efb814c6b8447ca8cfb6d2f2dc01031 P 4094f93fa463479dbaed35e8f6e5a437 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44919 } } peers { permanent_uuid: "4094f93fa463479dbaed35e8f6e5a437" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45101 } }
I20260926 09:27:47.567488 7751 ts_tablet_manager.cc:1428] T 9efb814c6b8447ca8cfb6d2f2dc01031 P 4094f93fa463479dbaed35e8f6e5a437: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:27:47.567891 7589 heartbeater.cc:503] Master 127.0.98.62:39437 was elected leader, sending a full tablet report...
I20260926 09:27:47.567919 7725 heartbeater.cc:503] Master 127.0.98.62:39437 was elected leader, sending a full tablet report...
I20260926 09:27:47.568893 7752 raft_consensus.cc:348] T 9efb814c6b8447ca8cfb6d2f2dc01031 P d7110ef2e34945f9958fbbcfec4c6f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44919 } } peers { permanent_uuid: "4094f93fa463479dbaed35e8f6e5a437" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45101 } }
I20260926 09:27:47.569049 7752 raft_consensus.cc:374] T 9efb814c6b8447ca8cfb6d2f2dc01031 P d7110ef2e34945f9958fbbcfec4c6f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:47.569092 7752 raft_consensus.cc:729] T 9efb814c6b8447ca8cfb6d2f2dc01031 P d7110ef2e34945f9958fbbcfec4c6f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7110ef2e34945f9958fbbcfec4c6f4d, State: Initialized, Role: FOLLOWER
I20260926 09:27:47.569199 7752 consensus_queue.cc:260] T 9efb814c6b8447ca8cfb6d2f2dc01031 P d7110ef2e34945f9958fbbcfec4c6f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44919 } } peers { permanent_uuid: "4094f93fa463479dbaed35e8f6e5a437" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45101 } }
I20260926 09:27:47.569465 7752 ts_tablet_manager.cc:1428] T 9efb814c6b8447ca8cfb6d2f2dc01031 P d7110ef2e34945f9958fbbcfec4c6f4d: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 09:27:47.569551 7452 heartbeater.cc:503] Master 127.0.98.62:39437 was elected leader, sending a full tablet report...
I20260926 09:27:47.634784 7756 raft_consensus.cc:482] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:27:47.634900 7756 raft_consensus.cc:504] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44919 } } peers { permanent_uuid: "4094f93fa463479dbaed35e8f6e5a437" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45101 } }
I20260926 09:27:47.635197 7756 leader_election.cc:288] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7110ef2e34945f9958fbbcfec4c6f4d (127.0.98.1:44919), 4094f93fa463479dbaed35e8f6e5a437 (127.0.98.2:45101)
I20260926 09:27:47.638223 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.635757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44161 (local address 127.0.98.2:45101)
0926 09:27:47.635808 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.635810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.635838 (+ 28us) server_negotiation.cc:407] Connection header received
0926 09:27:47.635864 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.635864 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.635876 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.635943 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:47.636083 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.636546 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.637405 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.637853 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.637945 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.637955 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.638103 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.638142 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 09:27:47.638224 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.635652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45101 (local address 127.0.98.3:44161)
0926 09:27:47.635737 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.635747 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:47.635803 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:47.635944 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:47.635945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:47.635946 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:47.636010 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.636012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.636639 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.636640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.637316 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.637317 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.637947 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.637948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.638021 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.638024 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:27:47.638099 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 09:27:47.638138 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 09:27:47.638242 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.635696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42029 (local address 127.0.98.1:44919)
0926 09:27:47.635752 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.635754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.635763 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:27:47.635792 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.635793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.635804 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.635854 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:47.636013 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.636479 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.637420 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.637875 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.637964 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.637974 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.638131 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.638167 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 09:27:47.638289 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.635527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44919 (local address 127.0.98.3:42029)
0926 09:27:47.635583 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.635594 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:47.635641 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:47.635858 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:47.635859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:47.635861 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:47.635933 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.635935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.636561 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.636562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.637315 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.637317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.637967 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.637968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.638043 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.638046 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:27:47.638124 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 09:27:47.638164 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":1}
I20260926 09:27:47.638784 7542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9efb814c6b8447ca8cfb6d2f2dc01031" candidate_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4094f93fa463479dbaed35e8f6e5a437" is_pre_election: true
I20260926 09:27:47.638784 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9efb814c6b8447ca8cfb6d2f2dc01031" candidate_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d" is_pre_election: true
I20260926 09:27:47.638945 7405 raft_consensus.cc:2454] T 9efb814c6b8447ca8cfb6d2f2dc01031 P d7110ef2e34945f9958fbbcfec4c6f4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9f71727a3f743c2b04bc0b854fdbad6 in term 0.
I20260926 09:27:47.638945 7542 raft_consensus.cc:2454] T 9efb814c6b8447ca8cfb6d2f2dc01031 P 4094f93fa463479dbaed35e8f6e5a437 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9f71727a3f743c2b04bc0b854fdbad6 in term 0.
I20260926 09:27:47.639187 7613 leader_election.cc:302] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9f71727a3f743c2b04bc0b854fdbad6, d7110ef2e34945f9958fbbcfec4c6f4d; no voters:
I20260926 09:27:47.639353 7756 raft_consensus.cc:2781] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:27:47.639432 7756 raft_consensus.cc:482] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:27:47.639456 7756 raft_consensus.cc:3037] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6 [term 0 FOLLOWER]: Advancing to term 1
W20260926 09:27:47.639962 7591 tablet.cc:2369] T 9efb814c6b8447ca8cfb6d2f2dc01031 P 4094f93fa463479dbaed35e8f6e5a437: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:27:47.640259 7756 raft_consensus.cc:504] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44919 } } peers { permanent_uuid: "4094f93fa463479dbaed35e8f6e5a437" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45101 } }
I20260926 09:27:47.640396 7756 leader_election.cc:288] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6 [CANDIDATE]: Term 1 election: Requested vote from peers d7110ef2e34945f9958fbbcfec4c6f4d (127.0.98.1:44919), 4094f93fa463479dbaed35e8f6e5a437 (127.0.98.2:45101)
I20260926 09:27:47.640609 7542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9efb814c6b8447ca8cfb6d2f2dc01031" candidate_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4094f93fa463479dbaed35e8f6e5a437"
I20260926 09:27:47.640604 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9efb814c6b8447ca8cfb6d2f2dc01031" candidate_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d"
I20260926 09:27:47.640693 7542 raft_consensus.cc:3037] T 9efb814c6b8447ca8cfb6d2f2dc01031 P 4094f93fa463479dbaed35e8f6e5a437 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:47.640694 7405 raft_consensus.cc:3037] T 9efb814c6b8447ca8cfb6d2f2dc01031 P d7110ef2e34945f9958fbbcfec4c6f4d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:47.641605 7405 raft_consensus.cc:2454] T 9efb814c6b8447ca8cfb6d2f2dc01031 P d7110ef2e34945f9958fbbcfec4c6f4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9f71727a3f743c2b04bc0b854fdbad6 in term 1.
I20260926 09:27:47.641602 7542 raft_consensus.cc:2454] T 9efb814c6b8447ca8cfb6d2f2dc01031 P 4094f93fa463479dbaed35e8f6e5a437 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9f71727a3f743c2b04bc0b854fdbad6 in term 1.
I20260926 09:27:47.641793 7613 leader_election.cc:302] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9f71727a3f743c2b04bc0b854fdbad6, d7110ef2e34945f9958fbbcfec4c6f4d; no voters:
I20260926 09:27:47.641877 7756 raft_consensus.cc:2781] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:47.642040 7756 raft_consensus.cc:686] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6 [term 1 LEADER]: Becoming Leader. State: Replica: a9f71727a3f743c2b04bc0b854fdbad6, State: Running, Role: LEADER
I20260926 09:27:47.642143 7756 consensus_queue.cc:237] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44919 } } peers { permanent_uuid: "4094f93fa463479dbaed35e8f6e5a437" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45101 } }
I20260926 09:27:47.642956 6487 catalog_manager.cc:5495] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9f71727a3f743c2b04bc0b854fdbad6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4094f93fa463479dbaed35e8f6e5a437" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45101 } health_report { overall_health: UNKNOWN } } }
I20260926 09:27:47.671015 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.668240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39395 (local address 127.0.0.1:43322)
0926 09:27:47.668313 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:47.668327 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:47.668396 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:47.668579 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:47.668580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:47.668582 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:47.668631 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.668633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.669442 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.669443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.670234 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:47.670236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.670796 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.670797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:47.670851 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.670854 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:27:47.670897 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 09:27:47.670939 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 09:27:47.671015 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.668324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43322 (local address 127.0.98.60:39395)
0926 09:27:47.668389 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.668391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.668400 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:27:47.668458 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.668458 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.668485 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.668571 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:47.668714 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.669318 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.670332 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.670716 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.670798 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.670876 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.670902 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.670940 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 09:27:47.671476 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.668872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43028 (local address 127.0.98.3:36285)
0926 09:27:47.668925 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.668928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.668948 (+ 20us) server_negotiation.cc:407] Connection header received
0926 09:27:47.668987 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.668988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.669016 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.669090 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:27:47.669235 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.669758 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.670497 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.670718 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.670793 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.670969 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:27:47.671102 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:27:47.671192 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.671375 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.671398 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 09:27:47.671440 7274 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.10] Dedup: 1.7->[1.8-1.10]
I20260926 09:27:47.672428 7311 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 705253badbc740e49445bb9ab7943235. Latest consensus state: current_term: 1 leader_uuid: "705253badbc740e49445bb9ab7943235" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: true } } }
I20260926 09:27:47.672540 7311 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [sys.catalog]: This master's current role is: LEARNER
I20260926 09:27:47.673347 7542 raft_consensus.cc:1261] T 9efb814c6b8447ca8cfb6d2f2dc01031 P 4094f93fa463479dbaed35e8f6e5a437 [term 1 FOLLOWER]: Refusing update from remote peer a9f71727a3f743c2b04bc0b854fdbad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:47.673385 7405 raft_consensus.cc:1261] T 9efb814c6b8447ca8cfb6d2f2dc01031 P d7110ef2e34945f9958fbbcfec4c6f4d [term 1 FOLLOWER]: Refusing update from remote peer a9f71727a3f743c2b04bc0b854fdbad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:47.673671 7756 consensus_queue.cc:1035] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4094f93fa463479dbaed35e8f6e5a437" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:47.673810 7759 consensus_queue.cc:1035] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:47.676545 7767 mvcc.cc:205] Tried to move back new op lower bound from 7333539297987371008 to 7333539297862811648. Current Snapshot: MvccSnapshot[applied={T|T < 7333539297987371008}]
I20260926 09:27:47.678261 7766 mvcc.cc:205] Tried to move back new op lower bound from 7333539297987371008 to 7333539297862811648. Current Snapshot: MvccSnapshot[applied={T|T < 7333539297987371008 or (T in {7333539297987371008})}]
I20260926 09:27:47.679654 6487 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:47.679757 6487 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:47.680280 7236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "705253badbc740e49445bb9ab7943235" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: true } } }
I20260926 09:27:47.680382 7236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:47.681006 6487 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235: Sending DeleteTablet for 3 replicas of tablet 9efb814c6b8447ca8cfb6d2f2dc01031
I20260926 09:27:47.681411 7385 tablet_service.cc:1514] Processing DeleteTablet for tablet 9efb814c6b8447ca8cfb6d2f2dc01031 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41130
I20260926 09:27:47.681393 7522 tablet_service.cc:1514] Processing DeleteTablet for tablet 9efb814c6b8447ca8cfb6d2f2dc01031 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40692
I20260926 09:27:47.681725 7047 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235: attempting to promote NON_VOTER ad655b5c635843ce8fa735418ada7abe to VOTER
I20260926 09:27:47.681855 7047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: false } }
I20260926 09:27:47.682243 7095 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [term 1 FOLLOWER]: Refusing update from remote peer 705253badbc740e49445bb9ab7943235: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20260926 09:27:47.682428 7282 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [term 1 LEARNER]: Refusing update from remote peer 705253badbc740e49445bb9ab7943235: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20260926 09:27:47.682436 7046 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260926 09:27:47.682730 7236 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260926 09:27:47.683182 7654 tablet_service.cc:1514] Processing DeleteTablet for tablet 9efb814c6b8447ca8cfb6d2f2dc01031 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43022
I20260926 09:27:47.683415 7775 tablet_replica.cc:330] T 9efb814c6b8447ca8cfb6d2f2dc01031 P d7110ef2e34945f9958fbbcfec4c6f4d: stopping tablet replica
I20260926 09:27:47.683418 7236 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 7 to 12, ad655b5c635843ce8fa735418ada7abe (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: false } } }
I20260926 09:27:47.683511 7775 raft_consensus.cc:2229] T 9efb814c6b8447ca8cfb6d2f2dc01031 P d7110ef2e34945f9958fbbcfec4c6f4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:47.683610 7775 raft_consensus.cc:2258] T 9efb814c6b8447ca8cfb6d2f2dc01031 P d7110ef2e34945f9958fbbcfec4c6f4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:47.683776 7776 tablet_replica.cc:330] T 9efb814c6b8447ca8cfb6d2f2dc01031 P 4094f93fa463479dbaed35e8f6e5a437: stopping tablet replica
I20260926 09:27:47.683858 7776 raft_consensus.cc:2229] T 9efb814c6b8447ca8cfb6d2f2dc01031 P 4094f93fa463479dbaed35e8f6e5a437 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:47.683934 7776 raft_consensus.cc:2258] T 9efb814c6b8447ca8cfb6d2f2dc01031 P 4094f93fa463479dbaed35e8f6e5a437 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:47.684376 7236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "705253badbc740e49445bb9ab7943235" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: false } } }
I20260926 09:27:47.684458 7236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:47.684506 7236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "705253badbc740e49445bb9ab7943235" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: false } } }
I20260926 09:27:47.684557 7236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:47.684801 7046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "705253badbc740e49445bb9ab7943235" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: false } } }
I20260926 09:27:47.684886 7046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 705253badbc740e49445bb9ab7943235 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:47.685057 7776 ts_tablet_manager.cc:1905] T 9efb814c6b8447ca8cfb6d2f2dc01031 P 4094f93fa463479dbaed35e8f6e5a437: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:47.685057 7282 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 7 to 12, ad655b5c635843ce8fa735418ada7abe (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: false } } }
I20260926 09:27:47.686113 7311 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "705253badbc740e49445bb9ab7943235" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: false } } }
I20260926 09:27:47.686197 7311 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:27:47.686529 7095 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 7 to 12, ad655b5c635843ce8fa735418ada7abe (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: false } } }
I20260926 09:27:47.686921 7776 ts_tablet_manager.cc:1918] T 9efb814c6b8447ca8cfb6d2f2dc01031 P 4094f93fa463479dbaed35e8f6e5a437: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:27:47.687031 7776 log.cc:1196] T 9efb814c6b8447ca8cfb6d2f2dc01031 P 4094f93fa463479dbaed35e8f6e5a437: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/wal/wals/9efb814c6b8447ca8cfb6d2f2dc01031
I20260926 09:27:47.687362 7776 ts_tablet_manager.cc:1939] T 9efb814c6b8447ca8cfb6d2f2dc01031 P 4094f93fa463479dbaed35e8f6e5a437: Deleting consensus metadata
I20260926 09:27:47.687824 6473 catalog_manager.cc:4841] TS 4094f93fa463479dbaed35e8f6e5a437 (127.0.98.2:45101): tablet 9efb814c6b8447ca8cfb6d2f2dc01031 (table test-table [id=b3a95e2536854cca8cf97eaadf106797]) successfully deleted
I20260926 09:27:47.688066 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 6455
I20260926 09:27:47.688369 7775 ts_tablet_manager.cc:1905] T 9efb814c6b8447ca8cfb6d2f2dc01031 P d7110ef2e34945f9958fbbcfec4c6f4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:47.688854 7779 sys_catalog.cc:455] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "705253badbc740e49445bb9ab7943235" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: false } } }
I20260926 09:27:47.688951 7779 sys_catalog.cc:458] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:27:47.689090 7778 tablet_replica.cc:330] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6: stopping tablet replica
I20260926 09:27:47.689186 7778 raft_consensus.cc:2229] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:27:47.689288 7778 raft_consensus.cc:2258] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:47.689729 7778 ts_tablet_manager.cc:1905] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:47.690135 7775 ts_tablet_manager.cc:1918] T 9efb814c6b8447ca8cfb6d2f2dc01031 P d7110ef2e34945f9958fbbcfec4c6f4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:27:47.690217 7775 log.cc:1196] T 9efb814c6b8447ca8cfb6d2f2dc01031 P d7110ef2e34945f9958fbbcfec4c6f4d: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-0/wal/wals/9efb814c6b8447ca8cfb6d2f2dc01031
I20260926 09:27:47.690533 7775 ts_tablet_manager.cc:1939] T 9efb814c6b8447ca8cfb6d2f2dc01031 P d7110ef2e34945f9958fbbcfec4c6f4d: Deleting consensus metadata
I20260926 09:27:47.690901 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.682016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43326 (local address 127.0.98.60:39395)
0926 09:27:47.682054 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.682056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.682091 (+ 35us) server_negotiation.cc:407] Connection header received
0926 09:27:47.682189 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.682190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.682200 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.682229 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:47.683184 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.683925 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.685329 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.686035 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.686082 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.687795 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:47.687796 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:47.687797 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:47.687804 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:47.689568 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.690028 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.690030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.690030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.690089 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.690330 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.690332 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.690332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.690489 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:47.690597 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.690730 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.690819 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:27:47.691489 7778 ts_tablet_manager.cc:1918] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:27:47.691579 7778 log.cc:1196] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/9efb814c6b8447ca8cfb6d2f2dc01031
I20260926 09:27:47.691812 7778 ts_tablet_manager.cc:1939] T 9efb814c6b8447ca8cfb6d2f2dc01031 P a9f71727a3f743c2b04bc0b854fdbad6: Deleting consensus metadata
W20260926 09:27:47.695148 7341 connection.cc:535] server connection from 127.0.0.1:41130 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 09:27:47.695245 7613 connection.cc:535] server connection from 127.0.0.1:43022 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 09:27:47.704281 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.696565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51640 (local address 127.0.98.61:38949)
0926 09:27:47.696613 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.696615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.696854 (+ 239us) server_negotiation.cc:407] Connection header received
0926 09:27:47.696910 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.696911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.696925 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.696956 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:47.697836 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.698604 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.699339 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.699633 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.699723 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.701135 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:47.701136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:47.701137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:47.701144 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:47.702938 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.703321 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.703323 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.703323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.703389 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.703613 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.703614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.703614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.703775 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:47.703890 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.704015 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.704112 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:27:47.705364 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:47.696687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43334 (local address 127.0.98.60:39395)
0926 09:27:47.696728 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:47.696730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:47.696882 (+ 152us) server_negotiation.cc:407] Connection header received
0926 09:27:47.696904 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:47.696904 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:47.696914 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:47.696976 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:47.698709 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.699461 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.700677 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:47.700975 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:47.701047 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:47.702410 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:47.702412 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:47.702412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:47.702418 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:47.704127 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.704515 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.704516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.704516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.704572 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:47.704776 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:47.704777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:47.704778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:47.704923 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:47.705019 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:47.705154 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 09:27:47.705226 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:27:48.677948 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:48.677632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.98.1:47869)
0926 09:27:48.677801 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:48.677884 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
W20260926 09:27:48.678224 7452 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39437 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
I20260926 09:27:48.678922 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:48.678659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.98.2:37841)
0926 09:27:48.678803 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:48.678855 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":60,"threads_started":1}
I20260926 09:27:48.679000 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:48.678713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.98.3:37343)
0926 09:27:48.678868 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:48.678931 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":53,"threads_started":1}
W20260926 09:27:48.679092 7589 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39437 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
W20260926 09:27:48.679263 7725 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39437 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
I20260926 09:27:49.186280 7790 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 705253badbc740e49445bb9ab7943235)
I20260926 09:27:49.186468 7790 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: false } }
I20260926 09:27:49.186750 7790 leader_election.cc:288] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 705253badbc740e49445bb9ab7943235 (127.0.98.62:39437), c056ade4d2f748619998a5e38bf29134 (127.0.98.61:38949)
I20260926 09:27:49.187101 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.186895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.0.1:56116)
0926 09:27:49.187005 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.187066 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":67,"threads_started":1}
W20260926 09:27:49.187268 7258 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
I20260926 09:27:49.187721 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.187612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.0.1:56126)
0926 09:27:49.187660 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.187691 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 09:27:49.187700 7795 raft_consensus.cc:482] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 705253badbc740e49445bb9ab7943235)
W20260926 09:27:49.187793 7258 leader_election.cc:334] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 705253badbc740e49445bb9ab7943235 (127.0.98.62:39437): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
I20260926 09:27:49.187901 7795 raft_consensus.cc:504] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: false } }
I20260926 09:27:49.188172 7795 leader_election.cc:288] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 705253badbc740e49445bb9ab7943235 (127.0.98.62:39437), ad655b5c635843ce8fa735418ada7abe (127.0.98.60:39395)
I20260926 09:27:49.188498 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.188241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.0.1:56134)
0926 09:27:49.188403 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.188455 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":55,"threads_started":1}
W20260926 09:27:49.188621 7071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
I20260926 09:27:49.189728 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.189635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.0.1:56138)
0926 09:27:49.189669 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.189701 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260926 09:27:49.189862 7071 leader_election.cc:334] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 705253badbc740e49445bb9ab7943235 (127.0.98.62:39437): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
I20260926 09:27:49.190415 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.187011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51642 (local address 127.0.98.61:38949)
0926 09:27:49.187115 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:49.187117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:49.187126 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:27:49.187153 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:49.187154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:49.187163 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:49.187226 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:49.187426 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:49.188158 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:49.189393 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:49.190073 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:49.190162 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:49.190174 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:49.190318 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 09:27:49.190357 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":87,"thread_start_us":54,"threads_started":1}
I20260926 09:27:49.190418 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.186908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38949 (local address 127.0.0.1:51642)
0926 09:27:49.187013 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.187024 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:49.187085 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:49.187253 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:49.187255 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:49.187257 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:49.187314 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:49.187317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:49.188270 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:49.188271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:49.189305 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:49.189308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:49.190163 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:49.190165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:49.190226 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:49.190230 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:27:49.190311 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 09:27:49.190353 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":49,"threads_started":1}
I20260926 09:27:49.190624 7095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad655b5c635843ce8fa735418ada7abe" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c056ade4d2f748619998a5e38bf29134" is_pre_election: true
I20260926 09:27:49.190742 7095 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad655b5c635843ce8fa735418ada7abe in term 1.
I20260926 09:27:49.190891 7256 leader_election.cc:302] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ad655b5c635843ce8fa735418ada7abe, c056ade4d2f748619998a5e38bf29134; no voters: 705253badbc740e49445bb9ab7943235
I20260926 09:27:49.191006 7790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 09:27:49.191046 7790 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [term 1 FOLLOWER]: Starting leader election (detected failure of leader 705253badbc740e49445bb9ab7943235)
I20260926 09:27:49.191078 7790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [term 1 FOLLOWER]: Advancing to term 2
I20260926 09:27:49.191510 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.188403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43350 (local address 127.0.98.60:39395)
0926 09:27:49.188588 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:49.188590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:49.188598 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:27:49.188620 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:49.188620 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:49.188631 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:49.188696 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:49.188848 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:49.189328 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:49.190673 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:49.191207 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:49.191273 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:49.191282 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:49.191427 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 09:27:49.191459 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":73,"threads_started":1}
I20260926 09:27:49.191519 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.188274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39395 (local address 127.0.0.1:43350)
0926 09:27:49.188427 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.188437 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:49.188487 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:49.188719 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:49.188721 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:49.188724 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:49.188765 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:49.188766 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:49.189439 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:49.189440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:49.190563 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:49.190567 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:49.191291 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:49.191292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:49.191350 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:49.191354 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:27:49.191415 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 09:27:49.191462 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":2,"thread_start_us":41,"threads_started":1}
I20260926 09:27:49.191962 7790 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [term 2 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: false } }
I20260926 09:27:49.192096 7790 leader_election.cc:288] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [CANDIDATE]: Term 2 election: Requested vote from peers 705253badbc740e49445bb9ab7943235 (127.0.98.62:39437), c056ade4d2f748619998a5e38bf29134 (127.0.98.61:38949)
I20260926 09:27:49.192270 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.192182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.0.1:56140)
0926 09:27:49.192219 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.192248 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 09:27:49.192277 7095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad655b5c635843ce8fa735418ada7abe" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c056ade4d2f748619998a5e38bf29134"
I20260926 09:27:49.192342 7095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [term 1 FOLLOWER]: Advancing to term 2
I20260926 09:27:49.192335 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c056ade4d2f748619998a5e38bf29134" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ad655b5c635843ce8fa735418ada7abe" is_pre_election: true
I20260926 09:27:49.192481 7282 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c056ade4d2f748619998a5e38bf29134 in current term 2: Already voted for candidate ad655b5c635843ce8fa735418ada7abe in this term.
I20260926 09:27:49.192566 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.192466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.0.1:56150)
0926 09:27:49.192518 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.192546 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 09:27:49.192607 7258 leader_election.cc:334] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 705253badbc740e49445bb9ab7943235 (127.0.98.62:39437): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
I20260926 09:27:49.192716 7072 leader_election.cc:302] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c056ade4d2f748619998a5e38bf29134; no voters: 705253badbc740e49445bb9ab7943235, ad655b5c635843ce8fa735418ada7abe
I20260926 09:27:49.193084 7095 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad655b5c635843ce8fa735418ada7abe in term 2.
I20260926 09:27:49.193162 7795 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:27:49.193270 7256 leader_election.cc:302] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ad655b5c635843ce8fa735418ada7abe, c056ade4d2f748619998a5e38bf29134; no voters: 705253badbc740e49445bb9ab7943235
I20260926 09:27:49.193377 7790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [term 2 FOLLOWER]: Leader election won for term 2
I20260926 09:27:49.193495 7790 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [term 2 LEADER]: Becoming Leader. State: Replica: ad655b5c635843ce8fa735418ada7abe, State: Running, Role: LEADER
I20260926 09:27:49.193552 7790 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: false } }
I20260926 09:27:49.193920 7800 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad655b5c635843ce8fa735418ada7abe. Latest consensus state: current_term: 2 leader_uuid: "ad655b5c635843ce8fa735418ada7abe" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: false } } }
I20260926 09:27:49.194021 7800 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:49.194191 7802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:27:49.194554 7802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:27:49.194607 7802 catalog_manager.cc:1259] Loaded cluster ID: 00db4655e98e4497b0dbe52a74a75983
I20260926 09:27:49.194622 7802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:27:49.194739 7802 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:27:49.194787 7802 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe: Loaded TSK: 0
I20260926 09:27:49.194873 7802 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:27:49.236897 7272 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:27:49.238627 7272 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:27:49.239607 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.239494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.0.1:56170)
0926 09:27:49.239546 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.239581 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 09:27:49.239626 7095 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [term 2 FOLLOWER]: Refusing update from remote peer ad655b5c635843ce8fa735418ada7abe: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 09:27:49.239895 7800 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [LEADER]: Connected to new peer: Peer: permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 09:27:49.240104 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.239983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.0.1:56186)
0926 09:27:49.240048 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.240078 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 09:27:49.240211 7258 consensus_peers.cc:487] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe -> Peer 705253badbc740e49445bb9ab7943235 (127.0.98.62:39437): Couldn't send request to peer 705253badbc740e49445bb9ab7943235. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 09:27:49.240528 7795 sys_catalog.cc:455] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad655b5c635843ce8fa735418ada7abe. Latest consensus state: current_term: 2 leader_uuid: "ad655b5c635843ce8fa735418ada7abe" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: false } } }
I20260926 09:27:49.240635 7795 sys_catalog.cc:458] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:27:49.240945 7803 mvcc.cc:205] Tried to move back new op lower bound from 7333539304403537920 to 7333539304217202688. Current Snapshot: MvccSnapshot[applied={T|T < 7333539304403537920}]
I20260926 09:27:49.241042 7800 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ad655b5c635843ce8fa735418ada7abe" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: false } } }
I20260926 09:27:49.241112 7800 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:49.241250 7795 sys_catalog.cc:455] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ad655b5c635843ce8fa735418ada7abe" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "705253badbc740e49445bb9ab7943235" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39437 } } peers { permanent_uuid: "c056ade4d2f748619998a5e38bf29134" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38949 } attrs { promote: false } } peers { permanent_uuid: "ad655b5c635843ce8fa735418ada7abe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39395 } attrs { promote: false } } }
I20260926 09:27:49.241358 7795 sys_catalog.cc:458] T 00000000000000000000000000000000 P c056ade4d2f748619998a5e38bf29134 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:27:49.241443 7804 mvcc.cc:205] Tried to move back new op lower bound from 7333539304403537920 to 7333539304217202688. Current Snapshot: MvccSnapshot[applied={T|T < 7333539304403537920 or (T in {7333539304403537920})}]
I20260926 09:27:49.246570 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.243109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44919 (local address 127.0.0.1:41140)
0926 09:27:49.243289 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.243301 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:49.243349 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:49.243660 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:49.243661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:49.243663 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:49.243691 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:49.243692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:49.244550 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:49.244550 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:49.245766 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:49.245768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:49.246314 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:49.246315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:49.246377 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:49.246381 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:27:49.246438 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 09:27:49.246503 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":38}
I20260926 09:27:49.246608 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.243390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41140 (local address 127.0.98.1:44919)
0926 09:27:49.243528 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:49.243530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:49.243538 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:27:49.243563 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:49.243564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:49.243573 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:49.243746 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:49.243780 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:49.244488 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:49.245853 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:49.246230 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:49.246315 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:49.246327 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:49.246462 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 09:27:49.246509 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20260926 09:27:49.247145 7385 tablet_service.cc:1467] Processing CreateTablet for tablet b0a5529dd3ef402fbc98c723a097d015 (DEFAULT_TABLE table=test-table [id=d72493887b294ddabda0b138c4d88baf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:49.247184 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.243202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45101 (local address 127.0.0.1:40696)
0926 09:27:49.243323 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.243331 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:49.243368 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:49.243792 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:49.243793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:49.243795 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:49.243849 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:49.243851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:49.244495 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:49.244496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:49.245766 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:49.245768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:49.246941 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:49.246942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:49.247011 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:49.247016 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:27:49.247074 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 09:27:49.247115 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":35,"thread_start_us":81,"threads_started":1}
I20260926 09:27:49.247306 7385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0a5529dd3ef402fbc98c723a097d015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:49.247200 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.243267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40696 (local address 127.0.98.2:45101)
0926 09:27:49.243644 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:49.243647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:49.243655 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:27:49.243678 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:49.243679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:49.243690 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:49.243748 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:49.243941 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:49.244412 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:49.246363 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:49.246843 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:49.246939 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:49.246970 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:49.247081 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 09:27:49.247128 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":50,"threads_started":1}
I20260926 09:27:49.247547 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.243092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36285 (local address 127.0.0.1:43036)
0926 09:27:49.243135 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.243146 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:49.243196 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:49.243519 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:49.243520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:49.243521 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:49.243571 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:49.243572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:49.244537 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:49.244538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:49.246512 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:49.246514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:49.247335 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:49.247336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:49.247398 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:49.247404 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 09:27:49.247436 (+ 32us) client_negotiation.cc:240] Negotiation successful
0926 09:27:49.247475 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":1226}
I20260926 09:27:49.247807 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.243154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43036 (local address 127.0.98.3:36285)
0926 09:27:49.243384 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:49.243386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:49.243394 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:27:49.243423 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:49.243424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:49.243437 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:49.243510 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:49.243658 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:49.244431 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:49.246606 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:49.247197 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:49.247649 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:49.247666 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:49.247690 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 09:27:49.247727 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":100,"threads_started":1}
I20260926 09:27:49.248040 7654 tablet_service.cc:1467] Processing CreateTablet for tablet b0a5529dd3ef402fbc98c723a097d015 (DEFAULT_TABLE table=test-table [id=d72493887b294ddabda0b138c4d88baf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:49.248183 7654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0a5529dd3ef402fbc98c723a097d015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:49.248327 7522 tablet_service.cc:1467] Processing CreateTablet for tablet b0a5529dd3ef402fbc98c723a097d015 (DEFAULT_TABLE table=test-table [id=d72493887b294ddabda0b138c4d88baf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:49.248435 7522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0a5529dd3ef402fbc98c723a097d015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:49.249069 7812 tablet_bootstrap.cc:492] T b0a5529dd3ef402fbc98c723a097d015 P d7110ef2e34945f9958fbbcfec4c6f4d: Bootstrap starting.
I20260926 09:27:49.249449 7812 tablet_bootstrap.cc:654] T b0a5529dd3ef402fbc98c723a097d015 P d7110ef2e34945f9958fbbcfec4c6f4d: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:49.249660 7813 tablet_bootstrap.cc:492] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6: Bootstrap starting.
I20260926 09:27:49.249919 7815 tablet_bootstrap.cc:492] T b0a5529dd3ef402fbc98c723a097d015 P 4094f93fa463479dbaed35e8f6e5a437: Bootstrap starting.
I20260926 09:27:49.250015 7813 tablet_bootstrap.cc:654] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:49.250144 7812 tablet_bootstrap.cc:492] T b0a5529dd3ef402fbc98c723a097d015 P d7110ef2e34945f9958fbbcfec4c6f4d: No bootstrap required, opened a new log
I20260926 09:27:49.250197 7812 ts_tablet_manager.cc:1397] T b0a5529dd3ef402fbc98c723a097d015 P d7110ef2e34945f9958fbbcfec4c6f4d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 09:27:49.250310 7812 raft_consensus.cc:348] T b0a5529dd3ef402fbc98c723a097d015 P d7110ef2e34945f9958fbbcfec4c6f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44919 } } peers { permanent_uuid: "4094f93fa463479dbaed35e8f6e5a437" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45101 } }
I20260926 09:27:49.250377 7812 raft_consensus.cc:374] T b0a5529dd3ef402fbc98c723a097d015 P d7110ef2e34945f9958fbbcfec4c6f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:49.250391 7812 raft_consensus.cc:729] T b0a5529dd3ef402fbc98c723a097d015 P d7110ef2e34945f9958fbbcfec4c6f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7110ef2e34945f9958fbbcfec4c6f4d, State: Initialized, Role: FOLLOWER
I20260926 09:27:49.250452 7812 consensus_queue.cc:260] T b0a5529dd3ef402fbc98c723a097d015 P d7110ef2e34945f9958fbbcfec4c6f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44919 } } peers { permanent_uuid: "4094f93fa463479dbaed35e8f6e5a437" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45101 } }
I20260926 09:27:49.250571 7813 tablet_bootstrap.cc:492] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6: No bootstrap required, opened a new log
I20260926 09:27:49.250607 7812 ts_tablet_manager.cc:1428] T b0a5529dd3ef402fbc98c723a097d015 P d7110ef2e34945f9958fbbcfec4c6f4d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 09:27:49.250630 7813 ts_tablet_manager.cc:1397] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:49.250743 7813 raft_consensus.cc:348] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44919 } } peers { permanent_uuid: "4094f93fa463479dbaed35e8f6e5a437" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45101 } }
I20260926 09:27:49.250830 7813 raft_consensus.cc:374] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:49.250854 7813 raft_consensus.cc:729] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f71727a3f743c2b04bc0b854fdbad6, State: Initialized, Role: FOLLOWER
I20260926 09:27:49.250912 7813 consensus_queue.cc:260] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44919 } } peers { permanent_uuid: "4094f93fa463479dbaed35e8f6e5a437" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45101 } }
I20260926 09:27:49.251045 7813 ts_tablet_manager.cc:1428] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:49.251104 7451 heartbeater.cc:503] Master 127.0.98.60:39395 was elected leader, sending a full tablet report...
I20260926 09:27:49.251273 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.250856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.98.1:57267)
0926 09:27:49.251188 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.251228 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":105,"threads_started":1}
I20260926 09:27:49.251647 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.251476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.98.1:36271)
0926 09:27:49.251590 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.251617 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260926 09:27:49.251770 7815 tablet_bootstrap.cc:654] T b0a5529dd3ef402fbc98c723a097d015 P 4094f93fa463479dbaed35e8f6e5a437: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:49.251937 7724 heartbeater.cc:503] Master 127.0.98.60:39395 was elected leader, sending a full tablet report...
I20260926 09:27:49.252177 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.251977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.98.3:40063)
0926 09:27:49.252093 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.252136 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":75,"threads_started":1}
I20260926 09:27:49.252523 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.252447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.98.3:34093)
0926 09:27:49.252477 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.252500 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260926 09:27:49.252678 7815 tablet_bootstrap.cc:492] T b0a5529dd3ef402fbc98c723a097d015 P 4094f93fa463479dbaed35e8f6e5a437: No bootstrap required, opened a new log
I20260926 09:27:49.252736 7815 ts_tablet_manager.cc:1397] T b0a5529dd3ef402fbc98c723a097d015 P 4094f93fa463479dbaed35e8f6e5a437: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 09:27:49.252876 7815 raft_consensus.cc:348] T b0a5529dd3ef402fbc98c723a097d015 P 4094f93fa463479dbaed35e8f6e5a437 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44919 } } peers { permanent_uuid: "4094f93fa463479dbaed35e8f6e5a437" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45101 } }
I20260926 09:27:49.252956 7815 raft_consensus.cc:374] T b0a5529dd3ef402fbc98c723a097d015 P 4094f93fa463479dbaed35e8f6e5a437 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:49.252988 7815 raft_consensus.cc:729] T b0a5529dd3ef402fbc98c723a097d015 P 4094f93fa463479dbaed35e8f6e5a437 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4094f93fa463479dbaed35e8f6e5a437, State: Initialized, Role: FOLLOWER
I20260926 09:27:49.253042 7815 consensus_queue.cc:260] T b0a5529dd3ef402fbc98c723a097d015 P 4094f93fa463479dbaed35e8f6e5a437 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44919 } } peers { permanent_uuid: "4094f93fa463479dbaed35e8f6e5a437" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45101 } }
I20260926 09:27:49.253201 7815 ts_tablet_manager.cc:1428] T b0a5529dd3ef402fbc98c723a097d015 P 4094f93fa463479dbaed35e8f6e5a437: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:49.253502 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.253320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.98.2:32803)
0926 09:27:49.253432 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.253467 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":85,"threads_started":1}
I20260926 09:27:49.253567 7588 heartbeater.cc:503] Master 127.0.98.60:39395 was elected leader, sending a full tablet report...
I20260926 09:27:49.254065 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.253960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.98.2:51569)
0926 09:27:49.254015 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.254039 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 09:27:49.260555 7454 tablet.cc:2369] T b0a5529dd3ef402fbc98c723a097d015 P d7110ef2e34945f9958fbbcfec4c6f4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:27:49.278005 7727 tablet.cc:2369] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:27:49.541265 7819 raft_consensus.cc:482] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:27:49.541373 7819 raft_consensus.cc:504] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44919 } } peers { permanent_uuid: "4094f93fa463479dbaed35e8f6e5a437" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45101 } }
I20260926 09:27:49.541561 7819 leader_election.cc:288] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7110ef2e34945f9958fbbcfec4c6f4d (127.0.98.1:44919), 4094f93fa463479dbaed35e8f6e5a437 (127.0.98.2:45101)
I20260926 09:27:49.541846 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0a5529dd3ef402fbc98c723a097d015" candidate_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d" is_pre_election: true
I20260926 09:27:49.541956 7405 raft_consensus.cc:2454] T b0a5529dd3ef402fbc98c723a097d015 P d7110ef2e34945f9958fbbcfec4c6f4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9f71727a3f743c2b04bc0b854fdbad6 in term 0.
I20260926 09:27:49.541895 7542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0a5529dd3ef402fbc98c723a097d015" candidate_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4094f93fa463479dbaed35e8f6e5a437" is_pre_election: true
I20260926 09:27:49.542014 7542 raft_consensus.cc:2454] T b0a5529dd3ef402fbc98c723a097d015 P 4094f93fa463479dbaed35e8f6e5a437 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9f71727a3f743c2b04bc0b854fdbad6 in term 0.
I20260926 09:27:49.542121 7613 leader_election.cc:302] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9f71727a3f743c2b04bc0b854fdbad6, d7110ef2e34945f9958fbbcfec4c6f4d; no voters:
I20260926 09:27:49.542248 7819 raft_consensus.cc:2781] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:27:49.542297 7819 raft_consensus.cc:482] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:27:49.542318 7819 raft_consensus.cc:3037] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:49.543517 7819 raft_consensus.cc:504] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44919 } } peers { permanent_uuid: "4094f93fa463479dbaed35e8f6e5a437" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45101 } }
I20260926 09:27:49.543646 7819 leader_election.cc:288] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6 [CANDIDATE]: Term 1 election: Requested vote from peers d7110ef2e34945f9958fbbcfec4c6f4d (127.0.98.1:44919), 4094f93fa463479dbaed35e8f6e5a437 (127.0.98.2:45101)
I20260926 09:27:49.543857 7542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0a5529dd3ef402fbc98c723a097d015" candidate_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4094f93fa463479dbaed35e8f6e5a437"
I20260926 09:27:49.543939 7542 raft_consensus.cc:3037] T b0a5529dd3ef402fbc98c723a097d015 P 4094f93fa463479dbaed35e8f6e5a437 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:49.544494 7542 raft_consensus.cc:2454] T b0a5529dd3ef402fbc98c723a097d015 P 4094f93fa463479dbaed35e8f6e5a437 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9f71727a3f743c2b04bc0b854fdbad6 in term 1.
I20260926 09:27:49.544622 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0a5529dd3ef402fbc98c723a097d015" candidate_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d"
I20260926 09:27:49.544678 7613 leader_election.cc:302] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4094f93fa463479dbaed35e8f6e5a437, a9f71727a3f743c2b04bc0b854fdbad6; no voters:
I20260926 09:27:49.544723 7405 raft_consensus.cc:3037] T b0a5529dd3ef402fbc98c723a097d015 P d7110ef2e34945f9958fbbcfec4c6f4d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:49.544844 7819 raft_consensus.cc:2781] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:49.545017 7819 raft_consensus.cc:686] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6 [term 1 LEADER]: Becoming Leader. State: Replica: a9f71727a3f743c2b04bc0b854fdbad6, State: Running, Role: LEADER
I20260926 09:27:49.545076 7819 consensus_queue.cc:237] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44919 } } peers { permanent_uuid: "4094f93fa463479dbaed35e8f6e5a437" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45101 } }
I20260926 09:27:49.545552 7405 raft_consensus.cc:2454] T b0a5529dd3ef402fbc98c723a097d015 P d7110ef2e34945f9958fbbcfec4c6f4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9f71727a3f743c2b04bc0b854fdbad6 in term 1.
I20260926 09:27:49.545631 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.545501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.98.3:50467)
0926 09:27:49.545542 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.545578 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 09:27:49.545951 7272 catalog_manager.cc:5495] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9f71727a3f743c2b04bc0b854fdbad6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f71727a3f743c2b04bc0b854fdbad6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4094f93fa463479dbaed35e8f6e5a437" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45101 } health_report { overall_health: UNKNOWN } } }
I20260926 09:27:49.600732 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.598023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43044 (local address 127.0.98.3:36285)
0926 09:27:49.598100 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:49.598102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:49.598114 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:27:49.598147 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:49.598148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:49.598162 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:49.598221 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:27:49.598534 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:49.599046 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:49.599846 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:49.600052 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:49.600109 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:49.600351 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:27:49.600385 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:27:49.600426 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:49.600635 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 09:27:49.600656 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 09:27:49.601902 7405 raft_consensus.cc:1261] T b0a5529dd3ef402fbc98c723a097d015 P d7110ef2e34945f9958fbbcfec4c6f4d [term 1 FOLLOWER]: Refusing update from remote peer a9f71727a3f743c2b04bc0b854fdbad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:49.601951 7542 raft_consensus.cc:1261] T b0a5529dd3ef402fbc98c723a097d015 P 4094f93fa463479dbaed35e8f6e5a437 [term 1 FOLLOWER]: Refusing update from remote peer a9f71727a3f743c2b04bc0b854fdbad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:49.602217 7819 consensus_queue.cc:1035] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4094f93fa463479dbaed35e8f6e5a437" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:49.602349 7825 consensus_queue.cc:1035] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7110ef2e34945f9958fbbcfec4c6f4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:49.603219 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.603094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.98.1:39855)
0926 09:27:49.603144 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.603177 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 09:27:49.603279 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.603161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.98.2:60613)
0926 09:27:49.603210 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.603241 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 09:27:49.605125 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.604709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.98.3:47271)
0926 09:27:49.605038 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.605091 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291}
I20260926 09:27:49.605818 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.605677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.98.2:40007)
0926 09:27:49.605716 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.605752 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260926 09:27:49.605993 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.605878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39437 (local address 127.0.98.1:43675)
0926 09:27:49.605913 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.605955 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 09:27:49.606329 7271 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:49.606433 7271 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:49.607893 7271 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad655b5c635843ce8fa735418ada7abe: Sending DeleteTablet for 3 replicas of tablet b0a5529dd3ef402fbc98c723a097d015
I20260926 09:27:49.608090 7385 tablet_service.cc:1514] Processing DeleteTablet for tablet b0a5529dd3ef402fbc98c723a097d015 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:49 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41140
I20260926 09:27:49.608295 7654 tablet_service.cc:1514] Processing DeleteTablet for tablet b0a5529dd3ef402fbc98c723a097d015 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:49 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43036
I20260926 09:27:49.608309 7522 tablet_service.cc:1514] Processing DeleteTablet for tablet b0a5529dd3ef402fbc98c723a097d015 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:49 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40696
I20260926 09:27:49.608536 7839 tablet_replica.cc:330] T b0a5529dd3ef402fbc98c723a097d015 P 4094f93fa463479dbaed35e8f6e5a437: stopping tablet replica
I20260926 09:27:49.608600 7839 raft_consensus.cc:2229] T b0a5529dd3ef402fbc98c723a097d015 P 4094f93fa463479dbaed35e8f6e5a437 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:49.608564 7838 tablet_replica.cc:330] T b0a5529dd3ef402fbc98c723a097d015 P d7110ef2e34945f9958fbbcfec4c6f4d: stopping tablet replica
I20260926 09:27:49.608564 7385 tablet_service.cc:1514] Processing DeleteTablet for tablet b0a5529dd3ef402fbc98c723a097d015 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:49 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41140
I20260926 09:27:49.608660 7839 raft_consensus.cc:2258] T b0a5529dd3ef402fbc98c723a097d015 P 4094f93fa463479dbaed35e8f6e5a437 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:49.608664 7838 raft_consensus.cc:2229] T b0a5529dd3ef402fbc98c723a097d015 P d7110ef2e34945f9958fbbcfec4c6f4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:49.608780 7840 tablet_replica.cc:330] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6: stopping tablet replica
I20260926 09:27:49.608857 7840 raft_consensus.cc:2229] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:27:49.608933 7840 raft_consensus.cc:2258] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:49.609170 7839 ts_tablet_manager.cc:1905] T b0a5529dd3ef402fbc98c723a097d015 P 4094f93fa463479dbaed35e8f6e5a437: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:49.609304 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 7233
I20260926 09:27:49.611222 7839 ts_tablet_manager.cc:1918] T b0a5529dd3ef402fbc98c723a097d015 P 4094f93fa463479dbaed35e8f6e5a437: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:27:49.611299 7839 log.cc:1196] T b0a5529dd3ef402fbc98c723a097d015 P 4094f93fa463479dbaed35e8f6e5a437: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-1/wal/wals/b0a5529dd3ef402fbc98c723a097d015
I20260926 09:27:49.611558 7839 ts_tablet_manager.cc:1939] T b0a5529dd3ef402fbc98c723a097d015 P 4094f93fa463479dbaed35e8f6e5a437: Deleting consensus metadata
I20260926 09:27:49.612049 7258 catalog_manager.cc:4841] TS 4094f93fa463479dbaed35e8f6e5a437 (127.0.98.2:45101): tablet b0a5529dd3ef402fbc98c723a097d015 (table test-table [id=d72493887b294ddabda0b138c4d88baf]) successfully deleted
I20260926 09:27:49.613821 7840 ts_tablet_manager.cc:1905] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:49.615803 7840 ts_tablet_manager.cc:1918] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:27:49.615886 7840 log.cc:1196] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/b0a5529dd3ef402fbc98c723a097d015
I20260926 09:27:49.616170 7840 ts_tablet_manager.cc:1939] T b0a5529dd3ef402fbc98c723a097d015 P a9f71727a3f743c2b04bc0b854fdbad6: Deleting consensus metadata
I20260926 09:27:49.616744 7257 catalog_manager.cc:4841] TS a9f71727a3f743c2b04bc0b854fdbad6 (127.0.98.3:36285): tablet b0a5529dd3ef402fbc98c723a097d015 (table test-table [id=d72493887b294ddabda0b138c4d88baf]) successfully deleted
I20260926 09:27:49.618270 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 7459
I20260926 09:27:49.618423 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:49.618272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44919 (local address 127.0.0.1:41146)
0926 09:27:49.618347 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:49.618376 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 09:27:49.618585 7258 catalog_manager.cc:4584] TS d7110ef2e34945f9958fbbcfec4c6f4d (127.0.98.1:44919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0a5529dd3ef402fbc98c723a097d015: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44919: connect: Connection refused (error 111)
I20260926 09:27:49.624346 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 7595
I20260926 09:27:49.629884 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 6951
I20260926 09:27:49.634878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 7139
2026-09-26T09:27:49Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4956 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:27:49 dist-test-slave-dfzm krb5kdc[7843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:27:49 dist-test-slave-dfzm krb5kdc[7843](info): set up 2 sockets
Sep 26 09:27:49 dist-test-slave-dfzm krb5kdc[7843](info): commencing operation
krb5kdc: starting...
W20260926 09:27:50.678112 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:27:50 dist-test-slave-dfzm krb5kdc[7843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:27:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:27:50Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 09:27:50.797271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37123
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:50.877429 7856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:50.877763 7856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:50.877807 7856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:50.879498 7856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:27:50.879572 7856 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 09:27:50.879595 7856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:50.879621 7856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:50.879655 7856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:27:50.879685 7856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:27:50.881193 7856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37123
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:50.881480 7856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:50.881748 7856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:50.886699 7864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:50.886711 7861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:50.886785 7856 server_base.cc:1034] running on GCE node
W20260926 09:27:50.886714 7862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:50.887276 7856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:50.887558 7856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:50.888712 7856 hybrid_clock.cc:648] HybridClock initialized: now 1790414870888699 us; error 30 us; skew 500 ppm
Sep 26 09:27:50 dist-test-slave-dfzm krb5kdc[7843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414870, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:50.891299 7856 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 09:27:50.891837 7856 webserver.cc:460] Webserver started at http://127.0.98.62:39841/ using document root <none> and password file <none>
I20260926 09:27:50.892076 7856 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:50.892124 7856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:50.892270 7856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:50.893362 7856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "938afe1d087541c79c8e79f6702a4c19"
format_stamp: "Formatted at 2026-09-26 09:27:50 on dist-test-slave-dfzm"
server_key: "c1f577d164501e4f84ad71292c05b783"
server_key_iv: "3e60562c9bbf0734a1c36a1f394722d7"
server_key_version: "encryptionkey@0"
I20260926 09:27:50.893764 7856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "938afe1d087541c79c8e79f6702a4c19"
format_stamp: "Formatted at 2026-09-26 09:27:50 on dist-test-slave-dfzm"
server_key: "c1f577d164501e4f84ad71292c05b783"
server_key_iv: "3e60562c9bbf0734a1c36a1f394722d7"
server_key_version: "encryptionkey@0"
I20260926 09:27:50.895097 7856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:50.895829 7871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:50.896000 7856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:50.896093 7856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "938afe1d087541c79c8e79f6702a4c19"
format_stamp: "Formatted at 2026-09-26 09:27:50 on dist-test-slave-dfzm"
server_key: "c1f577d164501e4f84ad71292c05b783"
server_key_iv: "3e60562c9bbf0734a1c36a1f394722d7"
server_key_version: "encryptionkey@0"
I20260926 09:27:50.896153 7856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:50.914467 7856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:50.916386 7856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:50.916496 7856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:50.921123 7856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37123
I20260926 09:27:50.921226 7933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37123 every 8 connection(s)
I20260926 09:27:50.921659 7856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:27:50.922273 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 7856
I20260926 09:27:50.922399 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:27:50.922494 7934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:50.922660 392 external_mini_cluster.cc:1428] Setting key ebdf5dfb4e7a3465ae875b03062f9da9
I20260926 09:27:50.925805 7934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19: Bootstrap starting.
I20260926 09:27:50.926559 7934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:50.926851 7934 log.cc:824] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:50.927595 7934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19: No bootstrap required, opened a new log
Sep 26 09:27:50 dist-test-slave-dfzm krb5kdc[7843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:27:50.930078 7934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938afe1d087541c79c8e79f6702a4c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37123 } }
I20260926 09:27:50.930280 7934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:50.930335 7934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 938afe1d087541c79c8e79f6702a4c19, State: Initialized, Role: FOLLOWER
I20260926 09:27:50.930481 7934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938afe1d087541c79c8e79f6702a4c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37123 } }
I20260926 09:27:50.930559 7934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:27:50.930583 7934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:27:50.930653 7934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:50.931399 7934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938afe1d087541c79c8e79f6702a4c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37123 } }
I20260926 09:27:50.931512 7934 leader_election.cc:302] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 938afe1d087541c79c8e79f6702a4c19; no voters:
I20260926 09:27:50.931687 7934 leader_election.cc:288] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:27:50.931725 7939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:50.931870 7939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19 [term 1 LEADER]: Becoming Leader. State: Replica: 938afe1d087541c79c8e79f6702a4c19, State: Running, Role: LEADER
I20260926 09:27:50.931968 7934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:27:50.932032 7939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938afe1d087541c79c8e79f6702a4c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37123 } }
I20260926 09:27:50.932665 7940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 938afe1d087541c79c8e79f6702a4c19. Latest consensus state: current_term: 1 leader_uuid: "938afe1d087541c79c8e79f6702a4c19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938afe1d087541c79c8e79f6702a4c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37123 } } }
I20260926 09:27:50.932670 7941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "938afe1d087541c79c8e79f6702a4c19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938afe1d087541c79c8e79f6702a4c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37123 } } }
I20260926 09:27:50.932772 7940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:50.932773 7941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:50.933037 7944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:27:50.933758 7944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:27:50.934783 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:50.923292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53370 (local address 127.0.98.62:37123)
0926 09:27:50.923448 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:50.923456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:50.923481 (+ 25us) server_negotiation.cc:407] Connection header received
0926 09:27:50.924008 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:50.924014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:50.924062 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:50.924180 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:50.924968 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:50.925966 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:50.927041 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:50.927580 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:50.927723 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:50.930307 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:50.930312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:50.930313 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:50.930326 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:50.932441 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:50.932985 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:50.932989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:50.932989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:50.933126 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:50.933401 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:50.933402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:50.933403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:50.933801 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:50.933978 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:50.934485 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 09:27:50.934603 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260926 09:27:50.935510 7944 catalog_manager.cc:1347] Generated new cluster ID: d8d7643e53ee4555b2aa6b20d2a51138
I20260926 09:27:50.935560 7944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:27:50.951826 7944 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:27:50.952579 7944 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:27:50.959309 7944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19: Generated new TSK 0
I20260926 09:27:50.959519 7944 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 09:27:50.988416 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37123
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:51.072702 7960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:51.072925 7960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:51.072953 7960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:51.074474 7960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:51.074571 7960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:51.074595 7960 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:27:51.074651 7960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:27:51.076223 7960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:51.076496 7960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:51.076738 7960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:51.081828 7965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:51.081833 7968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:51.081889 7966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:51.082084 7960 server_base.cc:1034] running on GCE node
I20260926 09:27:51.082278 7960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:51.082482 7960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:51.083639 7960 hybrid_clock.cc:648] HybridClock initialized: now 1790414871083591 us; error 50 us; skew 500 ppm
Sep 26 09:27:51 dist-test-slave-dfzm krb5kdc[7843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414871, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:51.086071 7960 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 09:27:51.086517 7960 webserver.cc:460] Webserver started at http://127.0.98.1:42511/ using document root <none> and password file <none>
I20260926 09:27:51.086763 7960 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:51.086833 7960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:51.086979 7960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:51.088034 7960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "14d8d8a3dc9e410eaaf247e5636ab762"
format_stamp: "Formatted at 2026-09-26 09:27:51 on dist-test-slave-dfzm"
server_key: "df2a0913b9c2afcd83576ee8d567a9a8"
server_key_iv: "2d7af9ccdbdab245e1bd347b2022b909"
server_key_version: "encryptionkey@0"
I20260926 09:27:51.088393 7960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "14d8d8a3dc9e410eaaf247e5636ab762"
format_stamp: "Formatted at 2026-09-26 09:27:51 on dist-test-slave-dfzm"
server_key: "df2a0913b9c2afcd83576ee8d567a9a8"
server_key_iv: "2d7af9ccdbdab245e1bd347b2022b909"
server_key_version: "encryptionkey@0"
I20260926 09:27:51.089769 7960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:27:51.090471 7975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:51.090646 7960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:51.090744 7960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "14d8d8a3dc9e410eaaf247e5636ab762"
format_stamp: "Formatted at 2026-09-26 09:27:51 on dist-test-slave-dfzm"
server_key: "df2a0913b9c2afcd83576ee8d567a9a8"
server_key_iv: "2d7af9ccdbdab245e1bd347b2022b909"
server_key_version: "encryptionkey@0"
I20260926 09:27:51.090827 7960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:51.102885 7960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:51.104887 7960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:51.105054 7960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:51.105652 7960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:51.105710 7960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:51.105754 7960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:51.105782 7960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:51.112740 7960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43171
Sep 26 09:27:51 dist-test-slave-dfzm krb5kdc[7843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414871, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:27:51.112814 8090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43171 every 8 connection(s)
I20260926 09:27:51.113118 7960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:27:51.113520 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 7960
I20260926 09:27:51.113669 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:27:51.113919 392 external_mini_cluster.cc:1428] Setting key f500233993e885e7a97d44c2ff4d8382
I20260926 09:27:51.116292 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.106120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58391 (local address 127.0.98.62:37123)
0926 09:27:51.106161 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:51.106163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:51.106383 (+ 220us) server_negotiation.cc:407] Connection header received
0926 09:27:51.106914 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:51.106916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:51.106929 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:51.106994 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:51.107831 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.108629 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.110431 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.110785 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.110856 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.113255 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:51.113256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:51.113257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:51.113264 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:51.114908 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:51.115320 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:51.115321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:51.115321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:51.115381 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:51.115628 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:51.115629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:51.115629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:51.115752 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:51.115823 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:51.116107 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 09:27:51.116200 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:27:51.116483 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.106128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37123 (local address 127.0.98.1:58391)
0926 09:27:51.106264 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:51.106298 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:51.106807 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:51.107069 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:51.107073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:51.107443 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:51.107730 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.107751 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.108748 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.108749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.110333 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.110336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.110866 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.110867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.111022 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.111694 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:51.111703 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:51.113148 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:51.114988 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:51.114996 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:51.114998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:51.115231 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:51.115452 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:51.115453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:51.115453 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:51.115560 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:51.115826 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:51.115833 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:51.115972 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 09:27:51.116076 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 09:27:51.116244 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":77,"threads_started":1}
I20260926 09:27:51.116942 8091 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37123
I20260926 09:27:51.117053 8091 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:51.117238 8091 heartbeater.cc:511] Master 127.0.98.62:37123 requested a full tablet report, sending...
I20260926 09:27:51.117748 7888 ts_manager.cc:194] Registered new tserver with Master: 14d8d8a3dc9e410eaaf247e5636ab762 (127.0.98.1:43171)
I20260926 09:27:51.118523 7888 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58391
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 09:27:51.124812 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.118920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35397 (local address 127.0.98.62:37123)
0926 09:27:51.118957 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:51.118959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:51.119189 (+ 230us) server_negotiation.cc:407] Connection header received
0926 09:27:51.119213 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:51.119214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:51.119225 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:51.119282 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:51.119550 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.120369 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.121075 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.121391 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.121472 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.122419 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:51.122421 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:51.122421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:51.122428 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:51.123846 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:51.124145 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:51.124146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:51.124146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:51.124232 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:51.124437 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:51.124438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:51.124438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:51.124534 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:51.124584 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:51.124694 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 09:27:51.124750 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 09:27:51.124828 8098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.118888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37123 (local address 127.0.98.1:35397)
0926 09:27:51.119068 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:51.119084 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:51.119142 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:51.119284 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:51.119285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:51.119430 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:51.119480 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.119481 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.120492 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.120493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.120994 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.120996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.121472 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.121472 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.121517 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.122048 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:51.122052 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:51.122330 (+ 278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:51.123916 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:51.123917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:51.123918 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:51.124082 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:51.124306 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:51.124307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:51.124307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:51.124376 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:51.124592 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:51.124593 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:51.124626 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 09:27:51.124677 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 09:27:51.124750 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":77,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 09:27:51.140878 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37123
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:51.222513 8100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:51.222743 8100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:51.222769 8100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:51.223883 8100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:51.223920 8100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:51.223941 8100 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:27:51.223958 8100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:27:51.225487 8100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:51.225766 8100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:51.225953 8100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:51.230944 8106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:51.230950 8108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:51.230965 8105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:51.231273 8100 server_base.cc:1034] running on GCE node
I20260926 09:27:51.231495 8100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:51.231779 8100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:51.232945 8100 hybrid_clock.cc:648] HybridClock initialized: now 1790414871232930 us; error 31 us; skew 500 ppm
Sep 26 09:27:51 dist-test-slave-dfzm krb5kdc[7843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414871, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:51.235641 8100 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 09:27:51.236117 8100 webserver.cc:460] Webserver started at http://127.0.98.2:44843/ using document root <none> and password file <none>
I20260926 09:27:51.236346 8100 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:51.236392 8100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:51.236555 8100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:51.237689 8100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "057eec7f37324574b965b69ae7eb1d27"
format_stamp: "Formatted at 2026-09-26 09:27:51 on dist-test-slave-dfzm"
server_key: "daed18307010983f1fb47b3af08272d2"
server_key_iv: "5a01a34b0915cc6e3b47e34dab7acfb3"
server_key_version: "encryptionkey@0"
I20260926 09:27:51.238057 8100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "057eec7f37324574b965b69ae7eb1d27"
format_stamp: "Formatted at 2026-09-26 09:27:51 on dist-test-slave-dfzm"
server_key: "daed18307010983f1fb47b3af08272d2"
server_key_iv: "5a01a34b0915cc6e3b47e34dab7acfb3"
server_key_version: "encryptionkey@0"
I20260926 09:27:51.239387 8100 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:27:51.240164 8115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:51.240360 8100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:27:51.240463 8100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "057eec7f37324574b965b69ae7eb1d27"
format_stamp: "Formatted at 2026-09-26 09:27:51 on dist-test-slave-dfzm"
server_key: "daed18307010983f1fb47b3af08272d2"
server_key_iv: "5a01a34b0915cc6e3b47e34dab7acfb3"
server_key_version: "encryptionkey@0"
I20260926 09:27:51.240537 8100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:51.256090 8100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:51.258304 8100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:51.258473 8100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:51.259065 8100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:51.259125 8100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:51.259159 8100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:51.259188 8100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 09:27:51 dist-test-slave-dfzm krb5kdc[7843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414871, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:27:51.266898 8100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42267
I20260926 09:27:51.266968 8230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42267 every 8 connection(s)
I20260926 09:27:51.267302 8100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:27:51.270020 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.259485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36405 (local address 127.0.98.62:37123)
0926 09:27:51.259565 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:51.259567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:51.259831 (+ 264us) server_negotiation.cc:407] Connection header received
0926 09:27:51.260608 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:51.260609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:51.260620 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:51.260719 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:51.261678 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.262333 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.264204 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.264422 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.264519 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.267455 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:51.267457 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:51.267457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:51.267467 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:51.268824 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:51.269230 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:51.269231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:51.269231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:51.269279 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:51.269460 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:51.269461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:51.269461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:51.269559 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:51.269669 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:51.269891 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 09:27:51.269958 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 09:27:51.270179 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.259475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37123 (local address 127.0.98.2:36405)
0926 09:27:51.259669 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:51.259720 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:51.260469 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:51.260802 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:51.260807 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:51.261232 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:51.261554 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.261561 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.262450 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.262451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.264097 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.264101 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.264538 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.264538 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.264707 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.265279 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:51.265292 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:51.267337 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:51.268904 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:51.268910 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:51.268911 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:51.269148 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:51.269322 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:51.269322 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:51.269322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:51.269411 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:51.269659 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:51.269664 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:51.269781 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 09:27:51.269867 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 09:27:51.269988 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":96,"threads_started":1}
I20260926 09:27:51.270669 8231 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37123
I20260926 09:27:51.270749 8231 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:51.270928 8231 heartbeater.cc:511] Master 127.0.98.62:37123 requested a full tablet report, sending...
I20260926 09:27:51.271271 7888 ts_manager.cc:194] Registered new tserver with Master: 057eec7f37324574b965b69ae7eb1d27 (127.0.98.2:42267)
I20260926 09:27:51.271649 7888 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36405
I20260926 09:27:51.275871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 8100
I20260926 09:27:51.276002 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:27:51.276258 392 external_mini_cluster.cc:1428] Setting key f0c7321a5a3ab215359e5110daa858f8
I20260926 09:27:51.277401 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.271767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33303 (local address 127.0.98.62:37123)
0926 09:27:51.271805 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:51.271807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:51.271956 (+ 149us) server_negotiation.cc:407] Connection header received
0926 09:27:51.271996 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:51.271997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:51.272007 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:51.272065 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:51.272413 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.272858 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.273561 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.273812 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.273866 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.274799 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:51.274800 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:51.274800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:51.274806 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:51.275972 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:51.276419 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:51.276434 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:51.276435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:51.276501 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:51.276760 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:51.276761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:51.276762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:51.276962 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:51.277034 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:51.277230 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 09:27:51.277311 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:27:51.277416 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.271694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37123 (local address 127.0.98.2:33303)
0926 09:27:51.271872 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:51.271882 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:51.271947 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:51.272074 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:51.272075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:51.272269 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:51.272342 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.272344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.272962 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.272963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.273486 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.273487 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.273878 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.273879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.273933 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.274414 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:51.274420 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:51.274725 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:51.276099 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:51.276100 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:51.276100 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:51.276338 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:51.276625 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:51.276626 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:51.276626 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:51.276683 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:51.277040 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:51.277041 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:51.277092 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 09:27:51.277199 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 09:27:51.277310 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 09:27:51.301043 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37123
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:51.386456 8240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:51.386682 8240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:51.386708 8240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:51.387827 8240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:51.387866 8240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:51.387887 8240 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:27:51.387905 8240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:27:51.389324 8240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:51.389523 8240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:51.389739 8240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:51.394917 8246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:51.394914 8248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:51.394910 8245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:51.395308 8240 server_base.cc:1034] running on GCE node
I20260926 09:27:51.395516 8240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:51.395808 8240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:51.396978 8240 hybrid_clock.cc:648] HybridClock initialized: now 1790414871396954 us; error 36 us; skew 500 ppm
Sep 26 09:27:51 dist-test-slave-dfzm krb5kdc[7843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414871, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:51.399657 8240 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 09:27:51.400187 8240 webserver.cc:460] Webserver started at http://127.0.98.3:42329/ using document root <none> and password file <none>
I20260926 09:27:51.400432 8240 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:51.400502 8240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:51.400665 8240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:51.401808 8240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "73c558247bb64d0c9eed0af9f7b6d310"
format_stamp: "Formatted at 2026-09-26 09:27:51 on dist-test-slave-dfzm"
server_key: "be834ab980a88d3f43aeecc9956479ec"
server_key_iv: "d1c1ad89ac633245e4d9480f7511742e"
server_key_version: "encryptionkey@0"
I20260926 09:27:51.402184 8240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "73c558247bb64d0c9eed0af9f7b6d310"
format_stamp: "Formatted at 2026-09-26 09:27:51 on dist-test-slave-dfzm"
server_key: "be834ab980a88d3f43aeecc9956479ec"
server_key_iv: "d1c1ad89ac633245e4d9480f7511742e"
server_key_version: "encryptionkey@0"
I20260926 09:27:51.403457 8240 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:27:51.404189 8255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:51.404354 8240 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:51.404453 8240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "73c558247bb64d0c9eed0af9f7b6d310"
format_stamp: "Formatted at 2026-09-26 09:27:51 on dist-test-slave-dfzm"
server_key: "be834ab980a88d3f43aeecc9956479ec"
server_key_iv: "d1c1ad89ac633245e4d9480f7511742e"
server_key_version: "encryptionkey@0"
I20260926 09:27:51.404536 8240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:51.425289 8240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:51.427821 8240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:51.428002 8240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:51.428540 8240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:51.428611 8240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:51.428656 8240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:51.428705 8240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 09:27:51 dist-test-slave-dfzm krb5kdc[7843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414871, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:27:51.436348 8240 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33271
I20260926 09:27:51.436465 8370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33271 every 8 connection(s)
I20260926 09:27:51.436756 8240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:27:51.439381 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.429131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40875 (local address 127.0.98.62:37123)
0926 09:27:51.429211 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:51.429215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:51.429399 (+ 184us) server_negotiation.cc:407] Connection header received
0926 09:27:51.430000 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:51.430002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:51.430015 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:51.430113 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:51.431024 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.431881 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.433117 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.433426 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.433517 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.436552 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:51.436554 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:51.436554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:51.436562 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:51.438113 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:51.438512 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:51.438513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:51.438513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:51.438564 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:51.438805 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:51.438806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:51.438806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:51.438908 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:51.438956 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:51.439227 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 09:27:51.439313 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 09:27:51.439456 8264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.429061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37123 (local address 127.0.98.3:40875)
0926 09:27:51.429259 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:51.429303 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:51.429895 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:51.430175 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:51.430178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:51.430585 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:51.430906 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.430914 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.432016 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.432018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.433041 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.433044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.433538 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.433539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.433704 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.434689 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:51.434697 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:51.436420 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:51.438202 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:51.438208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:51.438210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:51.438440 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:51.438624 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:51.438625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:51.438625 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:51.438748 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:51.438962 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:51.438966 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:51.439085 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 09:27:51.439168 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 09:27:51.439292 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":108,"threads_started":1}
I20260926 09:27:51.440042 8371 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37123
I20260926 09:27:51.440126 8371 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:51.440312 8371 heartbeater.cc:511] Master 127.0.98.62:37123 requested a full tablet report, sending...
I20260926 09:27:51.440675 7888 ts_manager.cc:194] Registered new tserver with Master: 73c558247bb64d0c9eed0af9f7b6d310 (127.0.98.3:33271)
I20260926 09:27:51.441149 7888 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40875
I20260926 09:27:51.446220 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 8240
I20260926 09:27:51.446308 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:27:51.446511 392 external_mini_cluster.cc:1428] Setting key 94a96093aa82a7156984c6e3bf4e53c6
I20260926 09:27:51.447043 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.441179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42645 (local address 127.0.98.62:37123)
0926 09:27:51.441232 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:51.441234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:51.441340 (+ 106us) server_negotiation.cc:407] Connection header received
0926 09:27:51.441412 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:51.441412 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:51.441422 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:51.441476 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:51.441861 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.442329 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.443263 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.443469 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.443550 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.444581 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:51.444582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:51.444582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:51.444590 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:51.445851 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:51.446224 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:51.446225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:51.446226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:51.446290 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:51.446500 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:51.446501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:51.446501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:51.446653 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:51.446712 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:51.446874 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 09:27:51.446951 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 09:27:51.447072 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.441112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37123 (local address 127.0.98.3:42645)
0926 09:27:51.441282 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:51.441293 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:51.441363 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:51.441482 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:51.441483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:51.441715 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:51.441776 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.441778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.442425 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.442426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.443109 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.443112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.443549 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.443550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.443620 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.444123 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:51.444128 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:51.444495 (+ 367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:51.445948 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:51.445949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:51.445950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:51.446149 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:51.446365 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:51.446366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:51.446367 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:51.446427 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:51.446715 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:51.446716 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:51.446761 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 09:27:51.446847 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 09:27:51.446953 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":83,"threads_started":1}
I20260926 09:27:51.447337 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 09:27:51 dist-test-slave-dfzm krb5kdc[7843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414871, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 09:27:51 dist-test-slave-dfzm krb5kdc[7843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:27:51 dist-test-slave-dfzm krb5kdc[7843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 09:27:51.468551 7937 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:51.463118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53384 (local address 127.0.98.62:37123)
0926 09:27:51.463178 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:51.463181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:51.463198 (+ 17us) server_negotiation.cc:407] Connection header received
0926 09:27:51.463223 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:51.463224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:51.463233 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:51.463292 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:51.463855 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.464340 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.465239 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.465420 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.465476 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.468446 (+ 2970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53384: BlockingRecv error: recv got EOF from 127.0.0.1:53384 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 09:27:51 dist-test-slave-dfzm krb5kdc[7843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414871, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:27:51.476245 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.469354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53396 (local address 127.0.98.62:37123)
0926 09:27:51.469423 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:51.469425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:51.469514 (+ 89us) server_negotiation.cc:407] Connection header received
0926 09:27:51.469643 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:51.469644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:51.469654 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:51.469701 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:51.470206 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.470746 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.471467 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.471698 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.471767 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.473920 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:51.473922 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:51.473922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:51.473930 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:51.475122 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:51.475459 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:51.475461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:51.475461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:51.475521 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:51.475717 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:51.475718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:51.475718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:51.475834 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:51.475941 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:51.476043 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 09:27:51.476122 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 09:27:51.477846 7888 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:27:51.479524 7888 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:27:51.486227 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.482126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.0.98.2:42267)
0926 09:27:51.482307 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:51.482312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:51.482375 (+ 63us) server_negotiation.cc:407] Connection header received
0926 09:27:51.482461 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:51.482463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:51.482480 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:51.482556 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:51.482774 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.483636 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.485209 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.485796 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.485894 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.485978 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:51.486079 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 09:27:51.486131 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20260926 09:27:51.486222 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.482010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42267 (local address 127.0.0.1:40074)
0926 09:27:51.482318 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:51.482328 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:51.482379 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:51.482633 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:51.482634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:51.482636 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:51.482694 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.482695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.483718 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.483719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.485109 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.485111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.485895 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.485896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.485972 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.485987 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 09:27:51.486063 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 09:27:51.486131 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":65,"thread_start_us":70,"threads_started":1}
I20260926 09:27:51.486243 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.481788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43171 (local address 127.0.0.1:51804)
0926 09:27:51.482098 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:51.482145 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:51.482207 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:51.482797 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:51.482798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:51.482799 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:51.482839 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.482841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.484030 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.484031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.484962 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.484964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.485948 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.485949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.486010 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.486014 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:27:51.486047 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 09:27:51.486124 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":87,"threads_started":1}
I20260926 09:27:51.486529 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.481903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51804 (local address 127.0.98.1:43171)
0926 09:27:51.482625 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:51.482629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:51.482639 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:27:51.482674 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:51.482675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:51.482690 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:51.482796 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:51.482900 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.483948 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.485059 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.485840 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.486283 (+ 443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.486380 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:51.486406 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 09:27:51.486442 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":109,"threads_started":1}
I20260926 09:27:51.486579 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.481783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33271 (local address 127.0.0.1:36092)
0926 09:27:51.482094 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:51.482149 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:51.482201 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:51.482508 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:51.482511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:51.482513 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:51.482581 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.482582 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.483880 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.483881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.485106 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.485109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.486087 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.486087 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.486138 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.486141 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:27:51.486189 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 09:27:51.486225 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":47,"thread_start_us":60,"threads_started":1}
I20260926 09:27:51.486969 8165 tablet_service.cc:1467] Processing CreateTablet for tablet 086ee6841cfa4496bd5e79514a02d4fd (DEFAULT_TABLE table=test-table [id=b3d6896ccd904d73a42471f198633e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:51.487335 8165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086ee6841cfa4496bd5e79514a02d4fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:51.487308 8025 tablet_service.cc:1467] Processing CreateTablet for tablet 086ee6841cfa4496bd5e79514a02d4fd (DEFAULT_TABLE table=test-table [id=b3d6896ccd904d73a42471f198633e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:51.487615 8025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086ee6841cfa4496bd5e79514a02d4fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:51.490336 8399 tablet_bootstrap.cc:492] T 086ee6841cfa4496bd5e79514a02d4fd P 057eec7f37324574b965b69ae7eb1d27: Bootstrap starting.
I20260926 09:27:51.490664 8400 tablet_bootstrap.cc:492] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762: Bootstrap starting.
I20260926 09:27:51.490995 8399 tablet_bootstrap.cc:654] T 086ee6841cfa4496bd5e79514a02d4fd P 057eec7f37324574b965b69ae7eb1d27: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:51.491187 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.481904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36092 (local address 127.0.98.3:33271)
0926 09:27:51.482348 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:51.482353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:51.482363 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:27:51.482403 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:51.482406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:51.482421 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:51.482507 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:51.482667 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.483798 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.485187 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.485918 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.485978 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.486055 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:51.486263 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 09:27:51.491065 (+ 4802us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":85,"threads_started":1}
I20260926 09:27:51.491261 8400 tablet_bootstrap.cc:654] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:51.491348 8399 log.cc:824] T 086ee6841cfa4496bd5e79514a02d4fd P 057eec7f37324574b965b69ae7eb1d27: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:51.491576 8400 log.cc:824] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:51.491839 8305 tablet_service.cc:1467] Processing CreateTablet for tablet 086ee6841cfa4496bd5e79514a02d4fd (DEFAULT_TABLE table=test-table [id=b3d6896ccd904d73a42471f198633e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:51.492170 8399 tablet_bootstrap.cc:492] T 086ee6841cfa4496bd5e79514a02d4fd P 057eec7f37324574b965b69ae7eb1d27: No bootstrap required, opened a new log
I20260926 09:27:51.492172 8305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086ee6841cfa4496bd5e79514a02d4fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:51.492251 8399 ts_tablet_manager.cc:1397] T 086ee6841cfa4496bd5e79514a02d4fd P 057eec7f37324574b965b69ae7eb1d27: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:27:51.492323 8400 tablet_bootstrap.cc:492] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762: No bootstrap required, opened a new log
I20260926 09:27:51.492388 8400 ts_tablet_manager.cc:1397] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:27:51.494225 8399 raft_consensus.cc:348] T 086ee6841cfa4496bd5e79514a02d4fd P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.494376 8399 raft_consensus.cc:374] T 086ee6841cfa4496bd5e79514a02d4fd P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:51.494419 8399 raft_consensus.cc:729] T 086ee6841cfa4496bd5e79514a02d4fd P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 057eec7f37324574b965b69ae7eb1d27, State: Initialized, Role: FOLLOWER
I20260926 09:27:51.494555 8399 consensus_queue.cc:260] T 086ee6841cfa4496bd5e79514a02d4fd P 057eec7f37324574b965b69ae7eb1d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.494660 8400 raft_consensus.cc:348] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.494817 8400 raft_consensus.cc:374] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:51.494851 8399 ts_tablet_manager.cc:1428] T 086ee6841cfa4496bd5e79514a02d4fd P 057eec7f37324574b965b69ae7eb1d27: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:27:51.494867 8400 raft_consensus.cc:729] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8d8a3dc9e410eaaf247e5636ab762, State: Initialized, Role: FOLLOWER
I20260926 09:27:51.494947 8231 heartbeater.cc:503] Master 127.0.98.62:37123 was elected leader, sending a full tablet report...
I20260926 09:27:51.495004 8400 consensus_queue.cc:260] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.495312 8400 ts_tablet_manager.cc:1428] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:27:51.495425 8091 heartbeater.cc:503] Master 127.0.98.62:37123 was elected leader, sending a full tablet report...
I20260926 09:27:51.496150 8405 tablet_bootstrap.cc:492] T 086ee6841cfa4496bd5e79514a02d4fd P 73c558247bb64d0c9eed0af9f7b6d310: Bootstrap starting.
I20260926 09:27:51.496780 8405 tablet_bootstrap.cc:654] T 086ee6841cfa4496bd5e79514a02d4fd P 73c558247bb64d0c9eed0af9f7b6d310: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:51.497061 8405 log.cc:824] T 086ee6841cfa4496bd5e79514a02d4fd P 73c558247bb64d0c9eed0af9f7b6d310: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:51.497844 8405 tablet_bootstrap.cc:492] T 086ee6841cfa4496bd5e79514a02d4fd P 73c558247bb64d0c9eed0af9f7b6d310: No bootstrap required, opened a new log
I20260926 09:27:51.497920 8405 ts_tablet_manager.cc:1397] T 086ee6841cfa4496bd5e79514a02d4fd P 73c558247bb64d0c9eed0af9f7b6d310: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:27:51.499346 8405 raft_consensus.cc:348] T 086ee6841cfa4496bd5e79514a02d4fd P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.499464 8405 raft_consensus.cc:374] T 086ee6841cfa4496bd5e79514a02d4fd P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:51.499486 8405 raft_consensus.cc:729] T 086ee6841cfa4496bd5e79514a02d4fd P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c558247bb64d0c9eed0af9f7b6d310, State: Initialized, Role: FOLLOWER
I20260926 09:27:51.499624 8405 consensus_queue.cc:260] T 086ee6841cfa4496bd5e79514a02d4fd P 73c558247bb64d0c9eed0af9f7b6d310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.499889 8405 ts_tablet_manager.cc:1428] T 086ee6841cfa4496bd5e79514a02d4fd P 73c558247bb64d0c9eed0af9f7b6d310: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:27:51.499902 8371 heartbeater.cc:503] Master 127.0.98.62:37123 was elected leader, sending a full tablet report...
W20260926 09:27:51.517931 8232 tablet.cc:2369] T 086ee6841cfa4496bd5e79514a02d4fd P 057eec7f37324574b965b69ae7eb1d27: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:27:51.614094 8092 tablet.cc:2369] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:27:51.618489 8404 raft_consensus.cc:482] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:27:51.618623 8404 raft_consensus.cc:504] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.618988 8404 leader_election.cc:288] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 057eec7f37324574b965b69ae7eb1d27 (127.0.98.2:42267), 73c558247bb64d0c9eed0af9f7b6d310 (127.0.98.3:33271)
I20260926 09:27:51.621970 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.619313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44767 (local address 127.0.98.2:42267)
0926 09:27:51.619358 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:51.619361 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:51.619431 (+ 70us) server_negotiation.cc:407] Connection header received
0926 09:27:51.619513 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:51.619514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:51.619526 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:51.619608 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:51.619767 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.620247 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.621255 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.621614 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.621703 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.621713 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:51.621860 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 09:27:51.621901 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:27:51.621989 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.619303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42779 (local address 127.0.98.3:33271)
0926 09:27:51.619358 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:51.619361 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:51.619489 (+ 128us) server_negotiation.cc:407] Connection header received
0926 09:27:51.619542 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:51.619543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:51.619555 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:51.619612 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:51.619818 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.620258 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.621218 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.621578 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.621702 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.621712 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:51.621893 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 09:27:51.621925 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:27:51.621997 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.619202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42267 (local address 127.0.98.1:44767)
0926 09:27:51.619344 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:51.619357 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:51.619430 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:51.619603 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:51.619603 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:51.619606 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:51.619685 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.619687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.620336 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.620337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.621128 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.621131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.621709 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.621710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.621789 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.621793 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:27:51.621836 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 09:27:51.621901 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260926 09:27:51.622007 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.619202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33271 (local address 127.0.98.1:42779)
0926 09:27:51.619383 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:51.619394 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:51.619482 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:51.619634 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:51.619635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:51.619637 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:51.619706 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.619707 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.620375 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.620375 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.621120 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.621122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.621713 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.621714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.621798 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.621801 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:27:51.621874 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 09:27:51.621917 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":2,"thread_start_us":75,"threads_started":1}
I20260926 09:27:51.622421 8325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086ee6841cfa4496bd5e79514a02d4fd" candidate_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c558247bb64d0c9eed0af9f7b6d310" is_pre_election: true
I20260926 09:27:51.622496 8185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086ee6841cfa4496bd5e79514a02d4fd" candidate_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "057eec7f37324574b965b69ae7eb1d27" is_pre_election: true
I20260926 09:27:51.622571 8325 raft_consensus.cc:2454] T 086ee6841cfa4496bd5e79514a02d4fd P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d8d8a3dc9e410eaaf247e5636ab762 in term 0.
I20260926 09:27:51.622659 8185 raft_consensus.cc:2454] T 086ee6841cfa4496bd5e79514a02d4fd P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d8d8a3dc9e410eaaf247e5636ab762 in term 0.
I20260926 09:27:51.622802 7977 leader_election.cc:302] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d8d8a3dc9e410eaaf247e5636ab762, 73c558247bb64d0c9eed0af9f7b6d310; no voters:
I20260926 09:27:51.622964 8404 raft_consensus.cc:2781] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:27:51.623008 8404 raft_consensus.cc:482] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:27:51.623023 8404 raft_consensus.cc:3037] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:51.623668 8404 raft_consensus.cc:504] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.623818 8404 leader_election.cc:288] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762 [CANDIDATE]: Term 1 election: Requested vote from peers 057eec7f37324574b965b69ae7eb1d27 (127.0.98.2:42267), 73c558247bb64d0c9eed0af9f7b6d310 (127.0.98.3:33271)
I20260926 09:27:51.623979 8185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086ee6841cfa4496bd5e79514a02d4fd" candidate_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "057eec7f37324574b965b69ae7eb1d27"
I20260926 09:27:51.624042 8185 raft_consensus.cc:3037] T 086ee6841cfa4496bd5e79514a02d4fd P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:51.624740 8185 raft_consensus.cc:2454] T 086ee6841cfa4496bd5e79514a02d4fd P 057eec7f37324574b965b69ae7eb1d27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8d8a3dc9e410eaaf247e5636ab762 in term 1.
I20260926 09:27:51.624861 8325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086ee6841cfa4496bd5e79514a02d4fd" candidate_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c558247bb64d0c9eed0af9f7b6d310"
I20260926 09:27:51.624923 7978 leader_election.cc:302] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 057eec7f37324574b965b69ae7eb1d27, 14d8d8a3dc9e410eaaf247e5636ab762; no voters:
I20260926 09:27:51.624945 8325 raft_consensus.cc:3037] T 086ee6841cfa4496bd5e79514a02d4fd P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:51.625108 8404 raft_consensus.cc:2781] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:51.625216 8404 raft_consensus.cc:686] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762 [term 1 LEADER]: Becoming Leader. State: Replica: 14d8d8a3dc9e410eaaf247e5636ab762, State: Running, Role: LEADER
I20260926 09:27:51.625319 8404 consensus_queue.cc:237] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.625876 8325 raft_consensus.cc:2454] T 086ee6841cfa4496bd5e79514a02d4fd P 73c558247bb64d0c9eed0af9f7b6d310 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8d8a3dc9e410eaaf247e5636ab762 in term 1.
I20260926 09:27:51.626250 7888 catalog_manager.cc:5495] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14d8d8a3dc9e410eaaf247e5636ab762 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } health_report { overall_health: UNKNOWN } } }
I20260926 09:27:51.666345 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.663750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51810 (local address 127.0.98.1:43171)
0926 09:27:51.663827 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:51.663829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:51.663836 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:27:51.663870 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:51.663871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:51.663890 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:51.663971 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:27:51.664129 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.664602 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.665431 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.665711 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.665800 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.665979 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:27:51.666097 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:27:51.666192 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:51.666217 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 09:27:51.666238 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 09:27:51.668210 8185 raft_consensus.cc:1261] T 086ee6841cfa4496bd5e79514a02d4fd P 057eec7f37324574b965b69ae7eb1d27 [term 1 FOLLOWER]: Refusing update from remote peer 14d8d8a3dc9e410eaaf247e5636ab762: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:51.668200 8325 raft_consensus.cc:1261] T 086ee6841cfa4496bd5e79514a02d4fd P 73c558247bb64d0c9eed0af9f7b6d310 [term 1 FOLLOWER]: Refusing update from remote peer 14d8d8a3dc9e410eaaf247e5636ab762: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:51.668597 8404 consensus_queue.cc:1035] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:51.668730 8410 consensus_queue.cc:1035] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:51.671203 8414 mvcc.cc:205] Tried to move back new op lower bound from 7333539314350006272 to 7333539314177933312. Current Snapshot: MvccSnapshot[applied={T|T < 7333539314350006272}]
I20260926 09:27:51.674201 7887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:51.674285 7887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:51.675064 7887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19: Sending DeleteTablet for 3 replicas of tablet 086ee6841cfa4496bd5e79514a02d4fd
I20260926 09:27:51.675405 8305 tablet_service.cc:1514] Processing DeleteTablet for tablet 086ee6841cfa4496bd5e79514a02d4fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36092
I20260926 09:27:51.675450 8025 tablet_service.cc:1514] Processing DeleteTablet for tablet 086ee6841cfa4496bd5e79514a02d4fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51804
I20260926 09:27:51.675503 8165 tablet_service.cc:1514] Processing DeleteTablet for tablet 086ee6841cfa4496bd5e79514a02d4fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40074
I20260926 09:27:51.675669 8423 tablet_replica.cc:330] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762: stopping tablet replica
I20260926 09:27:51.675786 8423 raft_consensus.cc:2229] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:27:51.675936 8424 tablet_replica.cc:330] T 086ee6841cfa4496bd5e79514a02d4fd P 057eec7f37324574b965b69ae7eb1d27: stopping tablet replica
I20260926 09:27:51.675935 8423 raft_consensus.cc:2258] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:51.675890 7887 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:27:51.676029 7887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:27:51.676046 8424 raft_consensus.cc:2229] T 086ee6841cfa4496bd5e79514a02d4fd P 057eec7f37324574b965b69ae7eb1d27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:51.676244 8424 raft_consensus.cc:2258] T 086ee6841cfa4496bd5e79514a02d4fd P 057eec7f37324574b965b69ae7eb1d27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:51.676430 8422 tablet_replica.cc:330] T 086ee6841cfa4496bd5e79514a02d4fd P 73c558247bb64d0c9eed0af9f7b6d310: stopping tablet replica
I20260926 09:27:51.676527 8422 raft_consensus.cc:2229] T 086ee6841cfa4496bd5e79514a02d4fd P 73c558247bb64d0c9eed0af9f7b6d310 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:51.676616 8422 raft_consensus.cc:2258] T 086ee6841cfa4496bd5e79514a02d4fd P 73c558247bb64d0c9eed0af9f7b6d310 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:51.677145 8422 ts_tablet_manager.cc:1905] T 086ee6841cfa4496bd5e79514a02d4fd P 73c558247bb64d0c9eed0af9f7b6d310: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:51.677388 8424 ts_tablet_manager.cc:1905] T 086ee6841cfa4496bd5e79514a02d4fd P 057eec7f37324574b965b69ae7eb1d27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:51.677644 8025 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6e1ac4c5e44f64b8409b0573fd64a0 (DEFAULT_TABLE table=test-table [id=d90e67a9f9514b9498e10a8583801240]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:51.677778 8025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6e1ac4c5e44f64b8409b0573fd64a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:51.679126 8400 tablet_bootstrap.cc:492] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 14d8d8a3dc9e410eaaf247e5636ab762: Bootstrap starting.
I20260926 09:27:51.679178 8165 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6e1ac4c5e44f64b8409b0573fd64a0 (DEFAULT_TABLE table=test-table [id=d90e67a9f9514b9498e10a8583801240]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:51.679352 8424 ts_tablet_manager.cc:1918] T 086ee6841cfa4496bd5e79514a02d4fd P 057eec7f37324574b965b69ae7eb1d27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:27:51.679481 8422 ts_tablet_manager.cc:1918] T 086ee6841cfa4496bd5e79514a02d4fd P 73c558247bb64d0c9eed0af9f7b6d310: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:27:51.679569 8422 log.cc:1196] T 086ee6841cfa4496bd5e79514a02d4fd P 73c558247bb64d0c9eed0af9f7b6d310: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/086ee6841cfa4496bd5e79514a02d4fd
I20260926 09:27:51.679587 8400 tablet_bootstrap.cc:654] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 14d8d8a3dc9e410eaaf247e5636ab762: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:51.679656 8424 log.cc:1196] T 086ee6841cfa4496bd5e79514a02d4fd P 057eec7f37324574b965b69ae7eb1d27: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-1/wal/wals/086ee6841cfa4496bd5e79514a02d4fd
I20260926 09:27:51.679659 8305 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6e1ac4c5e44f64b8409b0573fd64a0 (DEFAULT_TABLE table=test-table [id=d90e67a9f9514b9498e10a8583801240]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:51.679867 8305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6e1ac4c5e44f64b8409b0573fd64a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:51.680095 8422 ts_tablet_manager.cc:1939] T 086ee6841cfa4496bd5e79514a02d4fd P 73c558247bb64d0c9eed0af9f7b6d310: Deleting consensus metadata
I20260926 09:27:51.680208 8400 tablet_bootstrap.cc:492] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 14d8d8a3dc9e410eaaf247e5636ab762: No bootstrap required, opened a new log
I20260926 09:27:51.680262 8400 ts_tablet_manager.cc:1397] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 14d8d8a3dc9e410eaaf247e5636ab762: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:51.680367 8424 ts_tablet_manager.cc:1939] T 086ee6841cfa4496bd5e79514a02d4fd P 057eec7f37324574b965b69ae7eb1d27: Deleting consensus metadata
I20260926 09:27:51.680415 8400 raft_consensus.cc:348] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.680513 8400 raft_consensus.cc:374] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:51.680543 8400 raft_consensus.cc:729] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8d8a3dc9e410eaaf247e5636ab762, State: Initialized, Role: FOLLOWER
I20260926 09:27:51.680600 8400 consensus_queue.cc:260] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 14d8d8a3dc9e410eaaf247e5636ab762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.680704 8400 ts_tablet_manager.cc:1428] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 14d8d8a3dc9e410eaaf247e5636ab762: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:51.680958 7873 catalog_manager.cc:4841] TS 73c558247bb64d0c9eed0af9f7b6d310 (127.0.98.3:33271): tablet 086ee6841cfa4496bd5e79514a02d4fd (table test-table [id=b3d6896ccd904d73a42471f198633e80]) successfully deleted
I20260926 09:27:51.681073 8405 tablet_bootstrap.cc:492] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310: Bootstrap starting.
I20260926 09:27:51.681142 8165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6e1ac4c5e44f64b8409b0573fd64a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:51.681421 8405 tablet_bootstrap.cc:654] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:51.682014 8405 tablet_bootstrap.cc:492] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310: No bootstrap required, opened a new log
I20260926 09:27:51.682078 8405 ts_tablet_manager.cc:1397] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:51.682221 8405 raft_consensus.cc:348] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.682291 8405 raft_consensus.cc:374] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:51.682323 8405 raft_consensus.cc:729] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c558247bb64d0c9eed0af9f7b6d310, State: Initialized, Role: FOLLOWER
I20260926 09:27:51.682375 8405 consensus_queue.cc:260] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.682386 8399 tablet_bootstrap.cc:492] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27: Bootstrap starting.
I20260926 09:27:51.682464 8405 ts_tablet_manager.cc:1428] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:51.682767 8399 tablet_bootstrap.cc:654] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:51.683153 7874 catalog_manager.cc:4841] TS 057eec7f37324574b965b69ae7eb1d27 (127.0.98.2:42267): tablet 086ee6841cfa4496bd5e79514a02d4fd (table test-table [id=b3d6896ccd904d73a42471f198633e80]) successfully deleted
I20260926 09:27:51.684024 8399 tablet_bootstrap.cc:492] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27: No bootstrap required, opened a new log
I20260926 09:27:51.684077 8399 ts_tablet_manager.cc:1397] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:27:51.684216 8399 raft_consensus.cc:348] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.684288 8399 raft_consensus.cc:374] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:51.684309 8399 raft_consensus.cc:729] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 057eec7f37324574b965b69ae7eb1d27, State: Initialized, Role: FOLLOWER
I20260926 09:27:51.684350 8399 consensus_queue.cc:260] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.684429 8399 ts_tablet_manager.cc:1428] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:51.684834 8423 ts_tablet_manager.cc:1905] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:51.686398 8423 ts_tablet_manager.cc:1918] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:27:51.686474 8423 log.cc:1196] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-0/wal/wals/086ee6841cfa4496bd5e79514a02d4fd
I20260926 09:27:51.686710 8423 ts_tablet_manager.cc:1939] T 086ee6841cfa4496bd5e79514a02d4fd P 14d8d8a3dc9e410eaaf247e5636ab762: Deleting consensus metadata
I20260926 09:27:51.687119 7874 catalog_manager.cc:4841] TS 14d8d8a3dc9e410eaaf247e5636ab762 (127.0.98.1:43171): tablet 086ee6841cfa4496bd5e79514a02d4fd (table test-table [id=b3d6896ccd904d73a42471f198633e80]) successfully deleted
W20260926 09:27:51.687364 8372 tablet.cc:2369] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:27:51.702855 8403 raft_consensus.cc:482] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:27:51.702970 8403 raft_consensus.cc:504] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.703331 8403 leader_election.cc:288] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14d8d8a3dc9e410eaaf247e5636ab762 (127.0.98.1:43171), 73c558247bb64d0c9eed0af9f7b6d310 (127.0.98.3:33271)
I20260926 09:27:51.703851 8407 raft_consensus.cc:482] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:27:51.703928 8407 raft_consensus.cc:504] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.704239 8407 leader_election.cc:288] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14d8d8a3dc9e410eaaf247e5636ab762 (127.0.98.1:43171), 057eec7f37324574b965b69ae7eb1d27 (127.0.98.2:42267)
I20260926 09:27:51.707468 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.703500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43171 (local address 127.0.98.2:34125)
0926 09:27:51.703604 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:51.703616 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:51.703663 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:51.704019 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:51.704020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:51.704022 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:51.704092 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.704094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.705203 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.705204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.706472 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.706475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.707224 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.707225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.707297 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.707301 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:27:51.707345 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 09:27:51.707383 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":30}
I20260926 09:27:51.707767 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.703779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34125 (local address 127.0.98.1:43171)
0926 09:27:51.703853 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:51.703855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:51.703867 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:27:51.703895 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:51.703896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:51.703905 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:51.703964 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:51.704166 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.704919 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.706568 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.707105 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.707170 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.707180 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:51.707664 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 09:27:51.707705 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 09:27:51.707999 8045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6e1ac4c5e44f64b8409b0573fd64a0" candidate_uuid: "057eec7f37324574b965b69ae7eb1d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" is_pre_election: true
I20260926 09:27:51.708163 8045 raft_consensus.cc:2454] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 057eec7f37324574b965b69ae7eb1d27 in term 0.
I20260926 09:27:51.708374 8118 leader_election.cc:302] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 057eec7f37324574b965b69ae7eb1d27, 14d8d8a3dc9e410eaaf247e5636ab762; no voters:
I20260926 09:27:51.708498 8403 raft_consensus.cc:2781] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:27:51.708545 8403 raft_consensus.cc:482] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:27:51.708585 8403 raft_consensus.cc:3037] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:51.709115 8264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.704408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43171 (local address 127.0.98.3:40733)
0926 09:27:51.704620 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:51.704629 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:51.704663 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:51.705201 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:51.705202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:51.705204 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:51.705275 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.705277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.706290 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.706292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.707502 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.707505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.708897 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.708899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.708971 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.708974 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:27:51.709004 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 09:27:51.709052 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198}
I20260926 09:27:51.709359 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.705055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55795 (local address 127.0.98.2:42267)
0926 09:27:51.705091 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:51.705093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:51.705100 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:27:51.705118 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:51.705119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:51.705126 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:51.705173 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:51.705800 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.706486 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.708367 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.708978 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.709043 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.709052 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:51.709263 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 09:27:51.709297 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:27:51.709578 8403 raft_consensus.cc:504] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.709666 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.704490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42267 (local address 127.0.98.3:55795)
0926 09:27:51.704942 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:51.704951 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:51.704993 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:51.705620 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:51.705621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:51.705623 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:51.705703 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.705705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.706613 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.706614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.708270 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.708273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.709099 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.709100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.709184 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.709187 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:27:51.709473 (+ 286us) client_negotiation.cc:240] Negotiation successful
0926 09:27:51.709513 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":3,"thread_start_us":58,"threads_started":1}
I20260926 09:27:51.709743 8403 leader_election.cc:288] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27 [CANDIDATE]: Term 1 election: Requested vote from peers 14d8d8a3dc9e410eaaf247e5636ab762 (127.0.98.1:43171), 73c558247bb64d0c9eed0af9f7b6d310 (127.0.98.3:33271)
I20260926 09:27:51.709884 8185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6e1ac4c5e44f64b8409b0573fd64a0" candidate_uuid: "73c558247bb64d0c9eed0af9f7b6d310" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "057eec7f37324574b965b69ae7eb1d27" is_pre_election: true
I20260926 09:27:51.709976 8185 raft_consensus.cc:2379] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 73c558247bb64d0c9eed0af9f7b6d310 in current term 1: Already voted for candidate 057eec7f37324574b965b69ae7eb1d27 in this term.
I20260926 09:27:51.709939 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.704740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40733 (local address 127.0.98.1:43171)
0926 09:27:51.705065 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:51.705066 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:51.705072 (+ 6us) server_negotiation.cc:407] Connection header received
0926 09:27:51.705103 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:51.705104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:51.705112 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:51.705338 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:51.705378 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.706194 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.707937 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.708796 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.709567 (+ 771us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.709579 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:51.709618 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 09:27:51.709652 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":68,"threads_started":1}
I20260926 09:27:51.710083 8045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6e1ac4c5e44f64b8409b0573fd64a0" candidate_uuid: "057eec7f37324574b965b69ae7eb1d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8d8a3dc9e410eaaf247e5636ab762"
I20260926 09:27:51.710317 8045 raft_consensus.cc:3037] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:51.710284 8044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6e1ac4c5e44f64b8409b0573fd64a0" candidate_uuid: "73c558247bb64d0c9eed0af9f7b6d310" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" is_pre_election: true
I20260926 09:27:51.710572 8258 leader_election.cc:302] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c558247bb64d0c9eed0af9f7b6d310; no voters: 057eec7f37324574b965b69ae7eb1d27, 14d8d8a3dc9e410eaaf247e5636ab762
I20260926 09:27:51.710747 8407 raft_consensus.cc:3037] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:51.711107 8045 raft_consensus.cc:2454] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 14d8d8a3dc9e410eaaf247e5636ab762 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 057eec7f37324574b965b69ae7eb1d27 in term 1.
I20260926 09:27:51.711254 8118 leader_election.cc:302] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 057eec7f37324574b965b69ae7eb1d27, 14d8d8a3dc9e410eaaf247e5636ab762; no voters:
I20260926 09:27:51.711326 8403 raft_consensus.cc:2781] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:51.711450 8403 raft_consensus.cc:686] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27 [term 1 LEADER]: Becoming Leader. State: Replica: 057eec7f37324574b965b69ae7eb1d27, State: Running, Role: LEADER
I20260926 09:27:51.711512 8407 raft_consensus.cc:2726] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 09:27:51.711506 8403 consensus_queue.cc:237] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.711676 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.703738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35091 (local address 127.0.98.3:33271)
0926 09:27:51.703809 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:51.703811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:51.704027 (+ 216us) server_negotiation.cc:407] Connection header received
0926 09:27:51.704125 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:51.704126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:51.704134 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:51.704163 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:27:51.704416 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.705120 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.710780 (+ 5660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.711299 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.711344 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.711352 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:51.711582 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 09:27:51.711624 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 09:27:51.711891 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.703511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33271 (local address 127.0.98.2:35091)
0926 09:27:51.703959 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:51.703970 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:51.704053 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:51.704237 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:51.704238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:51.704240 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:27:51.704331 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.704333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.705223 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.705223 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.710683 (+ 5460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.710686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.711416 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.711417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.711492 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.711496 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:27:51.711774 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 09:27:51.711815 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":39,"thread_start_us":56,"threads_started":1}
I20260926 09:27:51.712266 8325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6e1ac4c5e44f64b8409b0573fd64a0" candidate_uuid: "057eec7f37324574b965b69ae7eb1d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c558247bb64d0c9eed0af9f7b6d310" is_pre_election: true
I20260926 09:27:51.712370 8325 raft_consensus.cc:2454] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 057eec7f37324574b965b69ae7eb1d27 in term 1.
I20260926 09:27:51.712277 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6e1ac4c5e44f64b8409b0573fd64a0" candidate_uuid: "057eec7f37324574b965b69ae7eb1d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c558247bb64d0c9eed0af9f7b6d310"
I20260926 09:27:51.712567 7888 catalog_manager.cc:5495] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 057eec7f37324574b965b69ae7eb1d27 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "057eec7f37324574b965b69ae7eb1d27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } health_report { overall_health: UNKNOWN } } }
I20260926 09:27:51.713101 8323 raft_consensus.cc:2454] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 057eec7f37324574b965b69ae7eb1d27 in term 1.
Sep 26 09:27:51 dist-test-slave-dfzm krb5kdc[7843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414870, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:27:51.746157 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.738412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53398 (local address 127.0.98.62:37123)
0926 09:27:51.738479 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:51.738481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:51.738586 (+ 105us) server_negotiation.cc:407] Connection header received
0926 09:27:51.738659 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:51.738660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:51.738673 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:51.738731 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:51.739215 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.739939 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.740898 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.741177 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.741260 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.743458 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:51.743459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:51.743459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:51.743467 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:51.744898 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:51.745328 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:51.745329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:51.745330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:51.745385 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:51.745609 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:51.745610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:51.745611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:51.745725 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:51.745797 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:51.745953 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 09:27:51.746035 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:27:51.746237 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:51.738330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37123 (local address 127.0.0.1:53398)
0926 09:27:51.738511 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:51.738522 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:51.738585 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:51.738764 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:51.738765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:51.739073 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:51.739123 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.739125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.740048 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.740049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.740808 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:51.740810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:51.741272 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:51.741274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:51.741316 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:51.741960 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:51.741964 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:51.743364 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:51.745000 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:51.745008 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:51.745010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:51.745216 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:51.745449 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:51.745450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:51.745450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:51.745505 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:51.745789 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:51.745795 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:51.745845 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 09:27:51.745932 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 09:27:51.746092 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":60,"thread_start_us":101,"threads_started":1}
I20260926 09:27:51.748176 7888 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53398:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 09:27:51.748374 7888 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:27:51.750213 8305 tablet_service.cc:1467] Processing CreateTablet for tablet 3b0ea6a87a03490baa482a75077fa3d9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=668452993f584c5a8c5c0da6ca2c30d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:27:51.750384 8305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b0ea6a87a03490baa482a75077fa3d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:51.751988 8405 tablet_bootstrap.cc:492] T 3b0ea6a87a03490baa482a75077fa3d9 P 73c558247bb64d0c9eed0af9f7b6d310: Bootstrap starting.
I20260926 09:27:51.752058 8165 tablet_service.cc:1467] Processing CreateTablet for tablet 3b0ea6a87a03490baa482a75077fa3d9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=668452993f584c5a8c5c0da6ca2c30d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:27:51.752205 8165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b0ea6a87a03490baa482a75077fa3d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:51.752415 8405 tablet_bootstrap.cc:654] T 3b0ea6a87a03490baa482a75077fa3d9 P 73c558247bb64d0c9eed0af9f7b6d310: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:51.753450 8405 tablet_bootstrap.cc:492] T 3b0ea6a87a03490baa482a75077fa3d9 P 73c558247bb64d0c9eed0af9f7b6d310: No bootstrap required, opened a new log
I20260926 09:27:51.753502 8405 ts_tablet_manager.cc:1397] T 3b0ea6a87a03490baa482a75077fa3d9 P 73c558247bb64d0c9eed0af9f7b6d310: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:27:51.753741 8399 tablet_bootstrap.cc:492] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27: Bootstrap starting.
I20260926 09:27:51.753849 8025 tablet_service.cc:1467] Processing CreateTablet for tablet 3b0ea6a87a03490baa482a75077fa3d9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=668452993f584c5a8c5c0da6ca2c30d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:27:51.753976 8025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b0ea6a87a03490baa482a75077fa3d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:51.753981 8405 raft_consensus.cc:348] T 3b0ea6a87a03490baa482a75077fa3d9 P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.754065 8405 raft_consensus.cc:374] T 3b0ea6a87a03490baa482a75077fa3d9 P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:51.754092 8405 raft_consensus.cc:729] T 3b0ea6a87a03490baa482a75077fa3d9 P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c558247bb64d0c9eed0af9f7b6d310, State: Initialized, Role: FOLLOWER
I20260926 09:27:51.754143 8405 consensus_queue.cc:260] T 3b0ea6a87a03490baa482a75077fa3d9 P 73c558247bb64d0c9eed0af9f7b6d310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.754190 8399 tablet_bootstrap.cc:654] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:51.754242 8405 ts_tablet_manager.cc:1428] T 3b0ea6a87a03490baa482a75077fa3d9 P 73c558247bb64d0c9eed0af9f7b6d310: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:51.754359 8407 tablet_replica.cc:439] T 3b0ea6a87a03490baa482a75077fa3d9 P 73c558247bb64d0c9eed0af9f7b6d310: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } } }
I20260926 09:27:51.754423 8407 tablet_replica.cc:442] T 3b0ea6a87a03490baa482a75077fa3d9 P 73c558247bb64d0c9eed0af9f7b6d310: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:27:51.755450 8400 tablet_bootstrap.cc:492] T 3b0ea6a87a03490baa482a75077fa3d9 P 14d8d8a3dc9e410eaaf247e5636ab762: Bootstrap starting.
I20260926 09:27:51.755873 8400 tablet_bootstrap.cc:654] T 3b0ea6a87a03490baa482a75077fa3d9 P 14d8d8a3dc9e410eaaf247e5636ab762: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:51.755904 8399 tablet_bootstrap.cc:492] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27: No bootstrap required, opened a new log
I20260926 09:27:51.755949 8399 ts_tablet_manager.cc:1397] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:27:51.756292 8399 raft_consensus.cc:348] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.756378 8399 raft_consensus.cc:374] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:51.756399 8399 raft_consensus.cc:729] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 057eec7f37324574b965b69ae7eb1d27, State: Initialized, Role: FOLLOWER
I20260926 09:27:51.756449 8399 consensus_queue.cc:260] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.756556 8399 ts_tablet_manager.cc:1428] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:51.756610 8431 tablet_replica.cc:439] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } } }
I20260926 09:27:51.756716 8431 tablet_replica.cc:442] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:27:51.757047 8400 tablet_bootstrap.cc:492] T 3b0ea6a87a03490baa482a75077fa3d9 P 14d8d8a3dc9e410eaaf247e5636ab762: No bootstrap required, opened a new log
I20260926 09:27:51.757102 8400 ts_tablet_manager.cc:1397] T 3b0ea6a87a03490baa482a75077fa3d9 P 14d8d8a3dc9e410eaaf247e5636ab762: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:27:51.757268 8400 raft_consensus.cc:348] T 3b0ea6a87a03490baa482a75077fa3d9 P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.757319 8400 raft_consensus.cc:374] T 3b0ea6a87a03490baa482a75077fa3d9 P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:51.757331 8400 raft_consensus.cc:729] T 3b0ea6a87a03490baa482a75077fa3d9 P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8d8a3dc9e410eaaf247e5636ab762, State: Initialized, Role: FOLLOWER
I20260926 09:27:51.757397 8400 consensus_queue.cc:260] T 3b0ea6a87a03490baa482a75077fa3d9 P 14d8d8a3dc9e410eaaf247e5636ab762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:51.757478 8400 ts_tablet_manager.cc:1428] T 3b0ea6a87a03490baa482a75077fa3d9 P 14d8d8a3dc9e410eaaf247e5636ab762: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:51.757568 8404 tablet_replica.cc:439] T 3b0ea6a87a03490baa482a75077fa3d9 P 14d8d8a3dc9e410eaaf247e5636ab762: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } } }
I20260926 09:27:51.757709 8404 tablet_replica.cc:442] T 3b0ea6a87a03490baa482a75077fa3d9 P 14d8d8a3dc9e410eaaf247e5636ab762: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:27:52.002578 8431 raft_consensus.cc:482] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:27:52.002677 8431 raft_consensus.cc:504] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:52.002882 8431 leader_election.cc:288] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14d8d8a3dc9e410eaaf247e5636ab762 (127.0.98.1:43171), 73c558247bb64d0c9eed0af9f7b6d310 (127.0.98.3:33271)
I20260926 09:27:52.003147 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b0ea6a87a03490baa482a75077fa3d9" candidate_uuid: "057eec7f37324574b965b69ae7eb1d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c558247bb64d0c9eed0af9f7b6d310" is_pre_election: true
I20260926 09:27:52.003146 8045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b0ea6a87a03490baa482a75077fa3d9" candidate_uuid: "057eec7f37324574b965b69ae7eb1d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" is_pre_election: true
I20260926 09:27:52.003247 8323 raft_consensus.cc:2454] T 3b0ea6a87a03490baa482a75077fa3d9 P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 057eec7f37324574b965b69ae7eb1d27 in term 0.
I20260926 09:27:52.003255 8045 raft_consensus.cc:2454] T 3b0ea6a87a03490baa482a75077fa3d9 P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 057eec7f37324574b965b69ae7eb1d27 in term 0.
I20260926 09:27:52.003410 8118 leader_election.cc:302] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 057eec7f37324574b965b69ae7eb1d27, 14d8d8a3dc9e410eaaf247e5636ab762; no voters:
I20260926 09:27:52.003548 8431 raft_consensus.cc:2781] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:27:52.003602 8431 raft_consensus.cc:482] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:27:52.003618 8431 raft_consensus.cc:3037] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:52.004323 8431 raft_consensus.cc:504] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:52.004478 8431 leader_election.cc:288] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27 [CANDIDATE]: Term 1 election: Requested vote from peers 14d8d8a3dc9e410eaaf247e5636ab762 (127.0.98.1:43171), 73c558247bb64d0c9eed0af9f7b6d310 (127.0.98.3:33271)
I20260926 09:27:52.004614 8045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b0ea6a87a03490baa482a75077fa3d9" candidate_uuid: "057eec7f37324574b965b69ae7eb1d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8d8a3dc9e410eaaf247e5636ab762"
I20260926 09:27:52.004690 8045 raft_consensus.cc:3037] T 3b0ea6a87a03490baa482a75077fa3d9 P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:52.004685 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b0ea6a87a03490baa482a75077fa3d9" candidate_uuid: "057eec7f37324574b965b69ae7eb1d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c558247bb64d0c9eed0af9f7b6d310"
I20260926 09:27:52.004765 8323 raft_consensus.cc:3037] T 3b0ea6a87a03490baa482a75077fa3d9 P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:52.005468 8045 raft_consensus.cc:2454] T 3b0ea6a87a03490baa482a75077fa3d9 P 14d8d8a3dc9e410eaaf247e5636ab762 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 057eec7f37324574b965b69ae7eb1d27 in term 1.
I20260926 09:27:52.005470 8323 raft_consensus.cc:2454] T 3b0ea6a87a03490baa482a75077fa3d9 P 73c558247bb64d0c9eed0af9f7b6d310 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 057eec7f37324574b965b69ae7eb1d27 in term 1.
I20260926 09:27:52.005694 8117 leader_election.cc:302] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 057eec7f37324574b965b69ae7eb1d27, 73c558247bb64d0c9eed0af9f7b6d310; no voters:
I20260926 09:27:52.005806 8431 raft_consensus.cc:2781] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:52.005880 8431 raft_consensus.cc:686] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27 [term 1 LEADER]: Becoming Leader. State: Replica: 057eec7f37324574b965b69ae7eb1d27, State: Running, Role: LEADER
I20260926 09:27:52.005932 8431 consensus_queue.cc:237] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:52.006197 8403 tablet_replica.cc:439] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27: TxnStatusTablet state changed. Reason: New leader 057eec7f37324574b965b69ae7eb1d27. Latest consensus state: current_term: 1 leader_uuid: "057eec7f37324574b965b69ae7eb1d27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } } }
I20260926 09:27:52.006297 8403 tablet_replica.cc:442] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:27:52.006476 8443 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 09:27:52.006507 7888 catalog_manager.cc:5495] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 057eec7f37324574b965b69ae7eb1d27 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "057eec7f37324574b965b69ae7eb1d27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } health_report { overall_health: UNKNOWN } } }
I20260926 09:27:52.006628 8443 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 09:27:52.038616 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:52.035855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.0.98.2:42267)
0926 09:27:52.035897 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:52.035899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:52.035966 (+ 67us) server_negotiation.cc:407] Connection header received
0926 09:27:52.036058 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:52.036059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:52.036075 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:52.036136 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:27:52.036287 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:52.036739 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:52.037837 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:52.038041 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:52.038101 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:52.038311 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:27:52.038408 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:27:52.038463 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:52.038540 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 09:27:52.038556 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:27:52.038687 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:52.035789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42267 (local address 127.0.0.1:40080)
0926 09:27:52.035916 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:52.035927 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:52.035979 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:52.036163 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:52.036164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:52.036165 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:27:52.036220 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:52.036222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:52.036811 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:52.036812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:52.037755 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:52.037759 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:52.038125 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:52.038126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:52.038199 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:52.038218 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:27:52.038472 (+ 254us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:27:52.038480 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 09:27:52.038538 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 09:27:52.038594 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 09:27:52.039959 8323 raft_consensus.cc:1261] T 3b0ea6a87a03490baa482a75077fa3d9 P 73c558247bb64d0c9eed0af9f7b6d310 [term 1 FOLLOWER]: Refusing update from remote peer 057eec7f37324574b965b69ae7eb1d27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:52.040014 8045 raft_consensus.cc:1261] T 3b0ea6a87a03490baa482a75077fa3d9 P 14d8d8a3dc9e410eaaf247e5636ab762 [term 1 FOLLOWER]: Refusing update from remote peer 057eec7f37324574b965b69ae7eb1d27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:52.040275 8403 consensus_queue.cc:1035] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:52.040414 8431 consensus_queue.cc:1035] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:52.041039 8407 tablet_replica.cc:439] T 3b0ea6a87a03490baa482a75077fa3d9 P 73c558247bb64d0c9eed0af9f7b6d310: TxnStatusTablet state changed. Reason: New leader 057eec7f37324574b965b69ae7eb1d27. Latest consensus state: current_term: 1 leader_uuid: "057eec7f37324574b965b69ae7eb1d27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } } }
I20260926 09:27:52.041147 8407 tablet_replica.cc:442] T 3b0ea6a87a03490baa482a75077fa3d9 P 73c558247bb64d0c9eed0af9f7b6d310: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:27:52.041102 8404 tablet_replica.cc:439] T 3b0ea6a87a03490baa482a75077fa3d9 P 14d8d8a3dc9e410eaaf247e5636ab762: TxnStatusTablet state changed. Reason: New leader 057eec7f37324574b965b69ae7eb1d27. Latest consensus state: current_term: 1 leader_uuid: "057eec7f37324574b965b69ae7eb1d27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } } }
I20260926 09:27:52.041189 8404 tablet_replica.cc:442] T 3b0ea6a87a03490baa482a75077fa3d9 P 14d8d8a3dc9e410eaaf247e5636ab762: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:27:52.041821 8431 tablet_replica.cc:439] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "057eec7f37324574b965b69ae7eb1d27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } } }
I20260926 09:27:52.041910 8431 tablet_replica.cc:442] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:27:52.041836 8403 tablet_replica.cc:439] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "057eec7f37324574b965b69ae7eb1d27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } } }
I20260926 09:27:52.042043 8403 tablet_replica.cc:442] T 3b0ea6a87a03490baa482a75077fa3d9 P 057eec7f37324574b965b69ae7eb1d27: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:27:52.042637 8415 mvcc.cc:205] Tried to move back new op lower bound from 7333539315873157120 to 7333539315736576000. Current Snapshot: MvccSnapshot[applied={T|T < 7333539315873157120 or (T in {7333539315873157120})}]
I20260926 09:27:52.042817 8407 tablet_replica.cc:439] T 3b0ea6a87a03490baa482a75077fa3d9 P 73c558247bb64d0c9eed0af9f7b6d310: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "057eec7f37324574b965b69ae7eb1d27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } } }
I20260926 09:27:52.042894 8407 tablet_replica.cc:442] T 3b0ea6a87a03490baa482a75077fa3d9 P 73c558247bb64d0c9eed0af9f7b6d310: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:27:52.043169 8413 mvcc.cc:205] Tried to move back new op lower bound from 7333539315873157120 to 7333539315736576000. Current Snapshot: MvccSnapshot[applied={T|T < 7333539315873157120}]
I20260926 09:27:52.043258 8404 tablet_replica.cc:439] T 3b0ea6a87a03490baa482a75077fa3d9 P 14d8d8a3dc9e410eaaf247e5636ab762: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "057eec7f37324574b965b69ae7eb1d27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } } }
I20260926 09:27:52.043332 8404 tablet_replica.cc:442] T 3b0ea6a87a03490baa482a75077fa3d9 P 14d8d8a3dc9e410eaaf247e5636ab762: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:27:52.046060 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:52.043455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.0.98.2:42267)
0926 09:27:52.043519 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:52.043521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:52.043529 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:27:52.043549 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:52.043550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:52.043559 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:52.043630 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:27:52.043833 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:52.044544 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:52.045345 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:52.045677 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:52.045750 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:52.045854 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:27:52.045886 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:27:52.045931 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:52.045974 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 09:27:52.045991 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:27:52.051064 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:52.048428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50427 (local address 127.0.98.2:42267)
0926 09:27:52.048463 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:52.048464 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:52.048629 (+ 165us) server_negotiation.cc:407] Connection header received
0926 09:27:52.048652 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:52.048652 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:52.048662 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:52.048710 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:27:52.048825 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:52.049495 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:52.050309 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:52.050598 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:52.050679 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:52.050839 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:27:52.050870 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:27:52.050922 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:52.050965 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 09:27:52.050983 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 09:27:52.051069 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:52.048368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42267 (local address 127.0.98.2:50427)
0926 09:27:52.048526 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:52.048535 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:52.048586 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:52.048718 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:52.048719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:52.048720 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:27:52.048768 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:52.048769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:52.049563 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:52.049563 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:52.050214 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:52.050216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:52.050699 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:52.050700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:52.050754 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:52.050765 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:27:52.050927 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:27:52.050929 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:27:52.050962 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 09:27:52.050993 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260926 09:27:52.053099 8323 raft_consensus.cc:1261] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310 [term 1 FOLLOWER]: Refusing update from remote peer 057eec7f37324574b965b69ae7eb1d27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:52.053207 8045 raft_consensus.cc:1261] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 14d8d8a3dc9e410eaaf247e5636ab762 [term 1 FOLLOWER]: Refusing update from remote peer 057eec7f37324574b965b69ae7eb1d27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:52.053363 8403 consensus_queue.cc:1035] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:52.053486 8448 consensus_queue.cc:1035] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:52.064767 7888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:52.064878 7888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:52.065495 7888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19: Sending DeleteTablet for 3 replicas of tablet 4a6e1ac4c5e44f64b8409b0573fd64a0
I20260926 09:27:52.065800 8305 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a6e1ac4c5e44f64b8409b0573fd64a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36092
I20260926 09:27:52.065840 8025 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a6e1ac4c5e44f64b8409b0573fd64a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51804
I20260926 09:27:52.065900 8422 tablet_replica.cc:330] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310: stopping tablet replica
I20260926 09:27:52.065935 8423 tablet_replica.cc:330] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 14d8d8a3dc9e410eaaf247e5636ab762: stopping tablet replica
I20260926 09:27:52.065948 8422 raft_consensus.cc:2229] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:52.065981 8422 raft_consensus.cc:2258] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:52.065985 8423 raft_consensus.cc:2229] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 14d8d8a3dc9e410eaaf247e5636ab762 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:52.066020 8423 raft_consensus.cc:2258] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 14d8d8a3dc9e410eaaf247e5636ab762 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:52.066273 8165 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a6e1ac4c5e44f64b8409b0573fd64a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40074
I20260926 09:27:52.066390 8424 tablet_replica.cc:330] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27: stopping tablet replica
I20260926 09:27:52.066396 8422 ts_tablet_manager.cc:1905] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:52.066453 8424 raft_consensus.cc:2229] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:27:52.066535 8424 raft_consensus.cc:2258] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:52.068252 8422 ts_tablet_manager.cc:1918] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:27:52.068310 8422 log.cc:1196] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/4a6e1ac4c5e44f64b8409b0573fd64a0
I20260926 09:27:52.068531 8422 ts_tablet_manager.cc:1939] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 73c558247bb64d0c9eed0af9f7b6d310: Deleting consensus metadata
I20260926 09:27:52.069207 8424 ts_tablet_manager.cc:1905] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:52.069281 8423 ts_tablet_manager.cc:1905] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 14d8d8a3dc9e410eaaf247e5636ab762: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:52.070794 8423 ts_tablet_manager.cc:1918] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 14d8d8a3dc9e410eaaf247e5636ab762: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:27:52.070848 8423 log.cc:1196] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 14d8d8a3dc9e410eaaf247e5636ab762: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-0/wal/wals/4a6e1ac4c5e44f64b8409b0573fd64a0
I20260926 09:27:52.071066 8423 ts_tablet_manager.cc:1939] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 14d8d8a3dc9e410eaaf247e5636ab762: Deleting consensus metadata
I20260926 09:27:52.071067 8424 ts_tablet_manager.cc:1918] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:27:52.071123 8424 log.cc:1196] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-1/wal/wals/4a6e1ac4c5e44f64b8409b0573fd64a0
I20260926 09:27:52.071348 8424 ts_tablet_manager.cc:1939] T 4a6e1ac4c5e44f64b8409b0573fd64a0 P 057eec7f37324574b965b69ae7eb1d27: Deleting consensus metadata
I20260926 09:27:52.071488 7874 catalog_manager.cc:4841] TS 14d8d8a3dc9e410eaaf247e5636ab762 (127.0.98.1:43171): tablet 4a6e1ac4c5e44f64b8409b0573fd64a0 (table test-table [id=d90e67a9f9514b9498e10a8583801240]) successfully deleted
I20260926 09:27:52.071637 7873 catalog_manager.cc:4841] TS 73c558247bb64d0c9eed0af9f7b6d310 (127.0.98.3:33271): tablet 4a6e1ac4c5e44f64b8409b0573fd64a0 (table test-table [id=d90e67a9f9514b9498e10a8583801240]) successfully deleted
I20260926 09:27:52.071789 7874 catalog_manager.cc:4841] TS 057eec7f37324574b965b69ae7eb1d27 (127.0.98.2:42267): tablet 4a6e1ac4c5e44f64b8409b0573fd64a0 (table test-table [id=d90e67a9f9514b9498e10a8583801240]) successfully deleted
Sep 26 09:27:52 dist-test-slave-dfzm krb5kdc[7843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 09:27:52 dist-test-slave-dfzm krb5kdc[7843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:27:52 dist-test-slave-dfzm krb5kdc[7843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 09:27:52.089248 7937 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:52.083622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.0.98.62:37123)
0926 09:27:52.083697 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:52.083699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:52.083824 (+ 125us) server_negotiation.cc:407] Connection header received
0926 09:27:52.083932 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:52.083933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:52.083944 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:52.084002 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:52.084560 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:52.085060 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:52.085932 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:52.086129 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:52.086215 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:52.089181 (+ 2966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53412: BlockingRecv error: recv got EOF from 127.0.0.1:53412 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 09:27:52 dist-test-slave-dfzm krb5kdc[7843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 09:27:52.099949 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:52.090119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53418 (local address 127.0.98.62:37123)
0926 09:27:52.090171 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:52.090173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:52.092875 (+ 2702us) server_negotiation.cc:407] Connection header received
0926 09:27:52.092954 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:52.092955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:52.092966 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:52.093030 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:52.093552 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:52.094165 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:52.094911 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:52.095110 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:52.095194 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:52.097332 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:52.097334 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:52.097334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:52.097342 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:52.098784 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:52.099135 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:52.099136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:52.099137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:52.099193 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:52.099415 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:52.099415 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:52.099416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:52.099527 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:52.099632 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:52.099733 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 09:27:52.099834 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 09:27:52.101449 7888 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:27:52.101656 7888 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:27:52.102979 8025 tablet_service.cc:1467] Processing CreateTablet for tablet 85e418ecc9e545d88b073835b07e7e2a (DEFAULT_TABLE table=test-table [id=fd411642f81e4885a23d3f3c521431af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:52.103031 8305 tablet_service.cc:1467] Processing CreateTablet for tablet 85e418ecc9e545d88b073835b07e7e2a (DEFAULT_TABLE table=test-table [id=fd411642f81e4885a23d3f3c521431af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:52.103024 8165 tablet_service.cc:1467] Processing CreateTablet for tablet 85e418ecc9e545d88b073835b07e7e2a (DEFAULT_TABLE table=test-table [id=fd411642f81e4885a23d3f3c521431af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:52.103118 8025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85e418ecc9e545d88b073835b07e7e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:52.103147 8165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85e418ecc9e545d88b073835b07e7e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:52.103147 8305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85e418ecc9e545d88b073835b07e7e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:52.104435 8405 tablet_bootstrap.cc:492] T 85e418ecc9e545d88b073835b07e7e2a P 73c558247bb64d0c9eed0af9f7b6d310: Bootstrap starting.
I20260926 09:27:52.104501 8400 tablet_bootstrap.cc:492] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762: Bootstrap starting.
I20260926 09:27:52.104775 8405 tablet_bootstrap.cc:654] T 85e418ecc9e545d88b073835b07e7e2a P 73c558247bb64d0c9eed0af9f7b6d310: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:52.104820 8400 tablet_bootstrap.cc:654] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:52.105085 8399 tablet_bootstrap.cc:492] T 85e418ecc9e545d88b073835b07e7e2a P 057eec7f37324574b965b69ae7eb1d27: Bootstrap starting.
I20260926 09:27:52.105456 8399 tablet_bootstrap.cc:654] T 85e418ecc9e545d88b073835b07e7e2a P 057eec7f37324574b965b69ae7eb1d27: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:52.105840 8400 tablet_bootstrap.cc:492] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762: No bootstrap required, opened a new log
I20260926 09:27:52.105901 8400 ts_tablet_manager.cc:1397] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:52.106024 8400 raft_consensus.cc:348] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:52.106091 8405 tablet_bootstrap.cc:492] T 85e418ecc9e545d88b073835b07e7e2a P 73c558247bb64d0c9eed0af9f7b6d310: No bootstrap required, opened a new log
I20260926 09:27:52.106105 8400 raft_consensus.cc:374] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:52.106127 8400 raft_consensus.cc:729] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8d8a3dc9e410eaaf247e5636ab762, State: Initialized, Role: FOLLOWER
I20260926 09:27:52.106133 8405 ts_tablet_manager.cc:1397] T 85e418ecc9e545d88b073835b07e7e2a P 73c558247bb64d0c9eed0af9f7b6d310: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:27:52.106179 8400 consensus_queue.cc:260] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:52.106276 8400 ts_tablet_manager.cc:1428] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:52.106281 8405 raft_consensus.cc:348] T 85e418ecc9e545d88b073835b07e7e2a P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:52.106356 8405 raft_consensus.cc:374] T 85e418ecc9e545d88b073835b07e7e2a P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:52.106384 8405 raft_consensus.cc:729] T 85e418ecc9e545d88b073835b07e7e2a P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c558247bb64d0c9eed0af9f7b6d310, State: Initialized, Role: FOLLOWER
I20260926 09:27:52.106420 8405 consensus_queue.cc:260] T 85e418ecc9e545d88b073835b07e7e2a P 73c558247bb64d0c9eed0af9f7b6d310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:52.106529 8405 ts_tablet_manager.cc:1428] T 85e418ecc9e545d88b073835b07e7e2a P 73c558247bb64d0c9eed0af9f7b6d310: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:52.106889 8399 tablet_bootstrap.cc:492] T 85e418ecc9e545d88b073835b07e7e2a P 057eec7f37324574b965b69ae7eb1d27: No bootstrap required, opened a new log
I20260926 09:27:52.106956 8399 ts_tablet_manager.cc:1397] T 85e418ecc9e545d88b073835b07e7e2a P 057eec7f37324574b965b69ae7eb1d27: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:27:52.107127 8399 raft_consensus.cc:348] T 85e418ecc9e545d88b073835b07e7e2a P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:52.107206 8399 raft_consensus.cc:374] T 85e418ecc9e545d88b073835b07e7e2a P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:52.107234 8399 raft_consensus.cc:729] T 85e418ecc9e545d88b073835b07e7e2a P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 057eec7f37324574b965b69ae7eb1d27, State: Initialized, Role: FOLLOWER
I20260926 09:27:52.107291 8399 consensus_queue.cc:260] T 85e418ecc9e545d88b073835b07e7e2a P 057eec7f37324574b965b69ae7eb1d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:52.107403 8399 ts_tablet_manager.cc:1428] T 85e418ecc9e545d88b073835b07e7e2a P 057eec7f37324574b965b69ae7eb1d27: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:52.196231 8404 raft_consensus.cc:482] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:27:52.196331 8404 raft_consensus.cc:504] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:52.196487 8404 leader_election.cc:288] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 057eec7f37324574b965b69ae7eb1d27 (127.0.98.2:42267), 73c558247bb64d0c9eed0af9f7b6d310 (127.0.98.3:33271)
I20260926 09:27:52.196732 8185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e418ecc9e545d88b073835b07e7e2a" candidate_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "057eec7f37324574b965b69ae7eb1d27" is_pre_election: true
I20260926 09:27:52.196787 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e418ecc9e545d88b073835b07e7e2a" candidate_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c558247bb64d0c9eed0af9f7b6d310" is_pre_election: true
I20260926 09:27:52.196869 8185 raft_consensus.cc:2454] T 85e418ecc9e545d88b073835b07e7e2a P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d8d8a3dc9e410eaaf247e5636ab762 in term 0.
I20260926 09:27:52.196895 8323 raft_consensus.cc:2454] T 85e418ecc9e545d88b073835b07e7e2a P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d8d8a3dc9e410eaaf247e5636ab762 in term 0.
I20260926 09:27:52.197033 7978 leader_election.cc:302] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 057eec7f37324574b965b69ae7eb1d27, 14d8d8a3dc9e410eaaf247e5636ab762; no voters:
I20260926 09:27:52.197152 8404 raft_consensus.cc:2781] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:27:52.197189 8404 raft_consensus.cc:482] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:27:52.197214 8404 raft_consensus.cc:3037] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:52.197992 8404 raft_consensus.cc:504] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:52.198173 8404 leader_election.cc:288] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762 [CANDIDATE]: Term 1 election: Requested vote from peers 057eec7f37324574b965b69ae7eb1d27 (127.0.98.2:42267), 73c558247bb64d0c9eed0af9f7b6d310 (127.0.98.3:33271)
I20260926 09:27:52.198347 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e418ecc9e545d88b073835b07e7e2a" candidate_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c558247bb64d0c9eed0af9f7b6d310"
I20260926 09:27:52.198408 8185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e418ecc9e545d88b073835b07e7e2a" candidate_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "057eec7f37324574b965b69ae7eb1d27"
I20260926 09:27:52.198457 8323 raft_consensus.cc:3037] T 85e418ecc9e545d88b073835b07e7e2a P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:52.198468 8185 raft_consensus.cc:3037] T 85e418ecc9e545d88b073835b07e7e2a P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:52.199185 8185 raft_consensus.cc:2454] T 85e418ecc9e545d88b073835b07e7e2a P 057eec7f37324574b965b69ae7eb1d27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8d8a3dc9e410eaaf247e5636ab762 in term 1.
I20260926 09:27:52.199239 8323 raft_consensus.cc:2454] T 85e418ecc9e545d88b073835b07e7e2a P 73c558247bb64d0c9eed0af9f7b6d310 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8d8a3dc9e410eaaf247e5636ab762 in term 1.
I20260926 09:27:52.199389 7977 leader_election.cc:302] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d8d8a3dc9e410eaaf247e5636ab762, 73c558247bb64d0c9eed0af9f7b6d310; no voters:
I20260926 09:27:52.199518 8404 raft_consensus.cc:2781] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:52.199635 8404 raft_consensus.cc:686] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762 [term 1 LEADER]: Becoming Leader. State: Replica: 14d8d8a3dc9e410eaaf247e5636ab762, State: Running, Role: LEADER
I20260926 09:27:52.199692 8404 consensus_queue.cc:237] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:52.200246 7888 catalog_manager.cc:5495] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14d8d8a3dc9e410eaaf247e5636ab762 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } health_report { overall_health: UNKNOWN } } }
I20260926 09:27:52.228605 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:52.226196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51818 (local address 127.0.98.1:43171)
0926 09:27:52.226330 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:52.226332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:52.226342 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:27:52.226369 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:52.226369 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:52.226379 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:52.226461 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:27:52.226618 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:52.227102 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:52.227852 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:52.228064 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:52.228141 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:52.228323 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:27:52.228364 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:27:52.228425 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:52.228486 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 09:27:52.228509 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
I20260926 09:27:52.229386 8185 raft_consensus.cc:1261] T 85e418ecc9e545d88b073835b07e7e2a P 057eec7f37324574b965b69ae7eb1d27 [term 1 FOLLOWER]: Refusing update from remote peer 14d8d8a3dc9e410eaaf247e5636ab762: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:52.229427 8323 raft_consensus.cc:1261] T 85e418ecc9e545d88b073835b07e7e2a P 73c558247bb64d0c9eed0af9f7b6d310 [term 1 FOLLOWER]: Refusing update from remote peer 14d8d8a3dc9e410eaaf247e5636ab762: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:52.229673 8404 consensus_queue.cc:1035] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:52.229820 8470 consensus_queue.cc:1035] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:52.232275 7888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:52.232368 7888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:52.233114 7888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19: Sending DeleteTablet for 3 replicas of tablet 85e418ecc9e545d88b073835b07e7e2a
I20260926 09:27:52.233392 8165 tablet_service.cc:1514] Processing DeleteTablet for tablet 85e418ecc9e545d88b073835b07e7e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40074
I20260926 09:27:52.233387 8305 tablet_service.cc:1514] Processing DeleteTablet for tablet 85e418ecc9e545d88b073835b07e7e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36092
I20260926 09:27:52.233479 8164 tablet_service.cc:1514] Processing DeleteTablet for tablet 85e418ecc9e545d88b073835b07e7e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40074
I20260926 09:27:52.233489 8422 tablet_replica.cc:330] T 85e418ecc9e545d88b073835b07e7e2a P 73c558247bb64d0c9eed0af9f7b6d310: stopping tablet replica
I20260926 09:27:52.233533 8422 raft_consensus.cc:2229] T 85e418ecc9e545d88b073835b07e7e2a P 73c558247bb64d0c9eed0af9f7b6d310 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:52.233541 8025 tablet_service.cc:1514] Processing DeleteTablet for tablet 85e418ecc9e545d88b073835b07e7e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51804
I20260926 09:27:52.233573 8422 raft_consensus.cc:2258] T 85e418ecc9e545d88b073835b07e7e2a P 73c558247bb64d0c9eed0af9f7b6d310 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:52.233637 8424 tablet_replica.cc:330] T 85e418ecc9e545d88b073835b07e7e2a P 057eec7f37324574b965b69ae7eb1d27: stopping tablet replica
I20260926 09:27:52.233693 8424 raft_consensus.cc:2229] T 85e418ecc9e545d88b073835b07e7e2a P 057eec7f37324574b965b69ae7eb1d27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:52.233736 8424 raft_consensus.cc:2258] T 85e418ecc9e545d88b073835b07e7e2a P 057eec7f37324574b965b69ae7eb1d27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:52.233888 8423 tablet_replica.cc:330] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762: stopping tablet replica
I20260926 09:27:52.233938 8423 raft_consensus.cc:2229] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:27:52.233990 8423 raft_consensus.cc:2258] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:52.233948 7887 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:27:52.234115 7887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:27:52.234330 8423 ts_tablet_manager.cc:1905] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:52.234658 8422 ts_tablet_manager.cc:1905] T 85e418ecc9e545d88b073835b07e7e2a P 73c558247bb64d0c9eed0af9f7b6d310: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:52.234676 8424 ts_tablet_manager.cc:1905] T 85e418ecc9e545d88b073835b07e7e2a P 057eec7f37324574b965b69ae7eb1d27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:52.235883 8423 ts_tablet_manager.cc:1918] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:27:52.235934 8423 log.cc:1196] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-0/wal/wals/85e418ecc9e545d88b073835b07e7e2a
I20260926 09:27:52.236074 8424 ts_tablet_manager.cc:1918] T 85e418ecc9e545d88b073835b07e7e2a P 057eec7f37324574b965b69ae7eb1d27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:27:52.236092 8422 ts_tablet_manager.cc:1918] T 85e418ecc9e545d88b073835b07e7e2a P 73c558247bb64d0c9eed0af9f7b6d310: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:27:52.236123 8424 log.cc:1196] T 85e418ecc9e545d88b073835b07e7e2a P 057eec7f37324574b965b69ae7eb1d27: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-1/wal/wals/85e418ecc9e545d88b073835b07e7e2a
I20260926 09:27:52.236142 8422 log.cc:1196] T 85e418ecc9e545d88b073835b07e7e2a P 73c558247bb64d0c9eed0af9f7b6d310: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/85e418ecc9e545d88b073835b07e7e2a
I20260926 09:27:52.236148 8423 ts_tablet_manager.cc:1939] T 85e418ecc9e545d88b073835b07e7e2a P 14d8d8a3dc9e410eaaf247e5636ab762: Deleting consensus metadata
I20260926 09:27:52.236363 8424 ts_tablet_manager.cc:1939] T 85e418ecc9e545d88b073835b07e7e2a P 057eec7f37324574b965b69ae7eb1d27: Deleting consensus metadata
I20260926 09:27:52.236358 8422 ts_tablet_manager.cc:1939] T 85e418ecc9e545d88b073835b07e7e2a P 73c558247bb64d0c9eed0af9f7b6d310: Deleting consensus metadata
I20260926 09:27:52.236771 7874 catalog_manager.cc:4841] TS 14d8d8a3dc9e410eaaf247e5636ab762 (127.0.98.1:43171): tablet 85e418ecc9e545d88b073835b07e7e2a (table test-table [id=fd411642f81e4885a23d3f3c521431af]) successfully deleted
I20260926 09:27:52.237010 7874 catalog_manager.cc:4841] TS 057eec7f37324574b965b69ae7eb1d27 (127.0.98.2:42267): tablet 85e418ecc9e545d88b073835b07e7e2a (table test-table [id=fd411642f81e4885a23d3f3c521431af]) successfully deleted
W20260926 09:27:52.237149 7874 catalog_manager.cc:4808] TS 057eec7f37324574b965b69ae7eb1d27 (127.0.98.2:42267): delete failed for tablet 85e418ecc9e545d88b073835b07e7e2a because the tablet was not found. No further retry: Not found: Tablet not found: 85e418ecc9e545d88b073835b07e7e2a
I20260926 09:27:52.237345 8305 tablet_service.cc:1467] Processing CreateTablet for tablet 57015949a0a848b9a6ad41b12e1b876f (DEFAULT_TABLE table=test-table [id=2944d4360747428e9f116af47549c3ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:52.237388 8025 tablet_service.cc:1467] Processing CreateTablet for tablet 57015949a0a848b9a6ad41b12e1b876f (DEFAULT_TABLE table=test-table [id=2944d4360747428e9f116af47549c3ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:52.237429 7873 catalog_manager.cc:4841] TS 73c558247bb64d0c9eed0af9f7b6d310 (127.0.98.3:33271): tablet 85e418ecc9e545d88b073835b07e7e2a (table test-table [id=fd411642f81e4885a23d3f3c521431af]) successfully deleted
I20260926 09:27:52.237470 8305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57015949a0a848b9a6ad41b12e1b876f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:52.237480 8025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57015949a0a848b9a6ad41b12e1b876f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:52.238417 8405 tablet_bootstrap.cc:492] T 57015949a0a848b9a6ad41b12e1b876f P 73c558247bb64d0c9eed0af9f7b6d310: Bootstrap starting.
I20260926 09:27:52.238646 8400 tablet_bootstrap.cc:492] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762: Bootstrap starting.
I20260926 09:27:52.238828 8405 tablet_bootstrap.cc:654] T 57015949a0a848b9a6ad41b12e1b876f P 73c558247bb64d0c9eed0af9f7b6d310: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:52.238955 8400 tablet_bootstrap.cc:654] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:52.239331 8164 tablet_service.cc:1467] Processing CreateTablet for tablet 57015949a0a848b9a6ad41b12e1b876f (DEFAULT_TABLE table=test-table [id=2944d4360747428e9f116af47549c3ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:27:52.239442 8164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57015949a0a848b9a6ad41b12e1b876f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:52.239472 8405 tablet_bootstrap.cc:492] T 57015949a0a848b9a6ad41b12e1b876f P 73c558247bb64d0c9eed0af9f7b6d310: No bootstrap required, opened a new log
I20260926 09:27:52.239521 8405 ts_tablet_manager.cc:1397] T 57015949a0a848b9a6ad41b12e1b876f P 73c558247bb64d0c9eed0af9f7b6d310: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:52.239672 8405 raft_consensus.cc:348] T 57015949a0a848b9a6ad41b12e1b876f P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:52.239753 8405 raft_consensus.cc:374] T 57015949a0a848b9a6ad41b12e1b876f P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:52.239787 8405 raft_consensus.cc:729] T 57015949a0a848b9a6ad41b12e1b876f P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c558247bb64d0c9eed0af9f7b6d310, State: Initialized, Role: FOLLOWER
I20260926 09:27:52.239847 8405 consensus_queue.cc:260] T 57015949a0a848b9a6ad41b12e1b876f P 73c558247bb64d0c9eed0af9f7b6d310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:52.239949 8405 ts_tablet_manager.cc:1428] T 57015949a0a848b9a6ad41b12e1b876f P 73c558247bb64d0c9eed0af9f7b6d310: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:52.240741 8399 tablet_bootstrap.cc:492] T 57015949a0a848b9a6ad41b12e1b876f P 057eec7f37324574b965b69ae7eb1d27: Bootstrap starting.
I20260926 09:27:52.241115 8399 tablet_bootstrap.cc:654] T 57015949a0a848b9a6ad41b12e1b876f P 057eec7f37324574b965b69ae7eb1d27: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:52.241123 8400 tablet_bootstrap.cc:492] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762: No bootstrap required, opened a new log
I20260926 09:27:52.241168 8400 ts_tablet_manager.cc:1397] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 09:27:52.241264 8400 raft_consensus.cc:348] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:52.241340 8400 raft_consensus.cc:374] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:52.241366 8400 raft_consensus.cc:729] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8d8a3dc9e410eaaf247e5636ab762, State: Initialized, Role: FOLLOWER
I20260926 09:27:52.241408 8400 consensus_queue.cc:260] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:52.241499 8400 ts_tablet_manager.cc:1428] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:52.241768 8399 tablet_bootstrap.cc:492] T 57015949a0a848b9a6ad41b12e1b876f P 057eec7f37324574b965b69ae7eb1d27: No bootstrap required, opened a new log
I20260926 09:27:52.241818 8399 ts_tablet_manager.cc:1397] T 57015949a0a848b9a6ad41b12e1b876f P 057eec7f37324574b965b69ae7eb1d27: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:52.241984 8399 raft_consensus.cc:348] T 57015949a0a848b9a6ad41b12e1b876f P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:52.242054 8399 raft_consensus.cc:374] T 57015949a0a848b9a6ad41b12e1b876f P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:52.242075 8399 raft_consensus.cc:729] T 57015949a0a848b9a6ad41b12e1b876f P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 057eec7f37324574b965b69ae7eb1d27, State: Initialized, Role: FOLLOWER
I20260926 09:27:52.242113 8399 consensus_queue.cc:260] T 57015949a0a848b9a6ad41b12e1b876f P 057eec7f37324574b965b69ae7eb1d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:52.242199 8399 ts_tablet_manager.cc:1428] T 57015949a0a848b9a6ad41b12e1b876f P 057eec7f37324574b965b69ae7eb1d27: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:52.288370 8470 raft_consensus.cc:482] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:27:52.288482 8470 raft_consensus.cc:504] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:52.288686 8470 leader_election.cc:288] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 057eec7f37324574b965b69ae7eb1d27 (127.0.98.2:42267), 73c558247bb64d0c9eed0af9f7b6d310 (127.0.98.3:33271)
I20260926 09:27:52.288880 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57015949a0a848b9a6ad41b12e1b876f" candidate_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c558247bb64d0c9eed0af9f7b6d310" is_pre_election: true
I20260926 09:27:52.288918 8185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57015949a0a848b9a6ad41b12e1b876f" candidate_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "057eec7f37324574b965b69ae7eb1d27" is_pre_election: true
I20260926 09:27:52.289001 8323 raft_consensus.cc:2454] T 57015949a0a848b9a6ad41b12e1b876f P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d8d8a3dc9e410eaaf247e5636ab762 in term 0.
I20260926 09:27:52.289038 8185 raft_consensus.cc:2454] T 57015949a0a848b9a6ad41b12e1b876f P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d8d8a3dc9e410eaaf247e5636ab762 in term 0.
I20260926 09:27:52.289179 7977 leader_election.cc:302] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d8d8a3dc9e410eaaf247e5636ab762, 73c558247bb64d0c9eed0af9f7b6d310; no voters:
I20260926 09:27:52.289291 8470 raft_consensus.cc:2781] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:27:52.289330 8470 raft_consensus.cc:482] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:27:52.289354 8470 raft_consensus.cc:3037] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:52.290158 8470 raft_consensus.cc:504] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:52.290345 8470 leader_election.cc:288] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762 [CANDIDATE]: Term 1 election: Requested vote from peers 057eec7f37324574b965b69ae7eb1d27 (127.0.98.2:42267), 73c558247bb64d0c9eed0af9f7b6d310 (127.0.98.3:33271)
I20260926 09:27:52.290459 8185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57015949a0a848b9a6ad41b12e1b876f" candidate_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "057eec7f37324574b965b69ae7eb1d27"
I20260926 09:27:52.290539 8185 raft_consensus.cc:3037] T 57015949a0a848b9a6ad41b12e1b876f P 057eec7f37324574b965b69ae7eb1d27 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:52.290503 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57015949a0a848b9a6ad41b12e1b876f" candidate_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c558247bb64d0c9eed0af9f7b6d310"
I20260926 09:27:52.290611 8323 raft_consensus.cc:3037] T 57015949a0a848b9a6ad41b12e1b876f P 73c558247bb64d0c9eed0af9f7b6d310 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:52.291091 8185 raft_consensus.cc:2454] T 57015949a0a848b9a6ad41b12e1b876f P 057eec7f37324574b965b69ae7eb1d27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8d8a3dc9e410eaaf247e5636ab762 in term 1.
I20260926 09:27:52.291258 8323 raft_consensus.cc:2454] T 57015949a0a848b9a6ad41b12e1b876f P 73c558247bb64d0c9eed0af9f7b6d310 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8d8a3dc9e410eaaf247e5636ab762 in term 1.
I20260926 09:27:52.291270 7978 leader_election.cc:302] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 057eec7f37324574b965b69ae7eb1d27, 14d8d8a3dc9e410eaaf247e5636ab762; no voters:
I20260926 09:27:52.291373 8470 raft_consensus.cc:2781] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:52.291435 8470 raft_consensus.cc:686] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762 [term 1 LEADER]: Becoming Leader. State: Replica: 14d8d8a3dc9e410eaaf247e5636ab762, State: Running, Role: LEADER
I20260926 09:27:52.291489 8470 consensus_queue.cc:237] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } }
I20260926 09:27:52.292091 7888 catalog_manager.cc:5495] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14d8d8a3dc9e410eaaf247e5636ab762 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8d8a3dc9e410eaaf247e5636ab762" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 } health_report { overall_health: UNKNOWN } } }
I20260926 09:27:52.305102 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:52.302735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33911 (local address 127.0.98.2:42267)
0926 09:27:52.302801 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:52.302803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:52.302896 (+ 93us) server_negotiation.cc:407] Connection header received
0926 09:27:52.302959 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:52.302960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:52.302971 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:52.303041 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:27:52.303144 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:52.303621 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:52.304378 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:52.304578 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:52.304663 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:52.304853 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:27:52.304891 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:27:52.304952 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:52.305013 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 09:27:52.305034 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 09:27:52.305137 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:52.302656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42267 (local address 127.0.98.1:33911)
0926 09:27:52.302824 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:52.302835 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:52.302897 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:52.303045 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:52.303045 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:52.303047 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:27:52.303094 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:52.303096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:52.303708 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:52.303709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:52.304290 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:52.304292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:52.304666 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:52.304667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:52.304736 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:52.304756 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:27:52.304954 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:27:52.304956 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:27:52.305013 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 09:27:52.305059 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260926 09:27:52.306902 8323 raft_consensus.cc:1261] T 57015949a0a848b9a6ad41b12e1b876f P 73c558247bb64d0c9eed0af9f7b6d310 [term 1 FOLLOWER]: Refusing update from remote peer 14d8d8a3dc9e410eaaf247e5636ab762: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:52.306943 8185 raft_consensus.cc:1261] T 57015949a0a848b9a6ad41b12e1b876f P 057eec7f37324574b965b69ae7eb1d27 [term 1 FOLLOWER]: Refusing update from remote peer 14d8d8a3dc9e410eaaf247e5636ab762: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:27:52.307108 8470 consensus_queue.cc:1035] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "057eec7f37324574b965b69ae7eb1d27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:52.307242 8404 consensus_queue.cc:1035] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c558247bb64d0c9eed0af9f7b6d310" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:27:52.313509 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:52.311363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49109 (local address 127.0.98.1:43171)
0926 09:27:52.311403 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:52.311407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:52.311420 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:27:52.311439 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:52.311440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:52.311447 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:52.311493 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:27:52.311626 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:52.312102 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:52.312828 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:52.313040 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:52.313114 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:52.313259 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:27:52.313293 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:27:52.313363 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:52.313414 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 09:27:52.313437 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:27:52.313530 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:52.311306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43171 (local address 127.0.98.2:49109)
0926 09:27:52.311344 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:52.311356 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:52.311395 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:52.311518 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:52.311519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:52.311521 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:27:52.311568 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:52.311570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:52.312188 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:52.312189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:52.312742 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:52.312743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:52.313110 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:52.313111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:52.313165 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:52.313171 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:27:52.313364 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:27:52.313367 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:27:52.313413 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 09:27:52.313464 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260926 09:27:52.319142 7887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:52.319239 7887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:27:52.319849 7887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 938afe1d087541c79c8e79f6702a4c19: Sending DeleteTablet for 3 replicas of tablet 57015949a0a848b9a6ad41b12e1b876f
I20260926 09:27:52.320072 8025 tablet_service.cc:1514] Processing DeleteTablet for tablet 57015949a0a848b9a6ad41b12e1b876f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51804
I20260926 09:27:52.320096 8164 tablet_service.cc:1514] Processing DeleteTablet for tablet 57015949a0a848b9a6ad41b12e1b876f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40074
I20260926 09:27:52.320142 8305 tablet_service.cc:1514] Processing DeleteTablet for tablet 57015949a0a848b9a6ad41b12e1b876f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:27:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36092
I20260926 09:27:52.320191 8423 tablet_replica.cc:330] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762: stopping tablet replica
I20260926 09:27:52.320199 8424 tablet_replica.cc:330] T 57015949a0a848b9a6ad41b12e1b876f P 057eec7f37324574b965b69ae7eb1d27: stopping tablet replica
I20260926 09:27:52.320228 8422 tablet_replica.cc:330] T 57015949a0a848b9a6ad41b12e1b876f P 73c558247bb64d0c9eed0af9f7b6d310: stopping tablet replica
I20260926 09:27:52.320247 8423 raft_consensus.cc:2229] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:27:52.320247 8424 raft_consensus.cc:2229] T 57015949a0a848b9a6ad41b12e1b876f P 057eec7f37324574b965b69ae7eb1d27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:52.320281 8424 raft_consensus.cc:2258] T 57015949a0a848b9a6ad41b12e1b876f P 057eec7f37324574b965b69ae7eb1d27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:52.320287 8422 raft_consensus.cc:2229] T 57015949a0a848b9a6ad41b12e1b876f P 73c558247bb64d0c9eed0af9f7b6d310 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:27:52.320300 8423 raft_consensus.cc:2258] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:52.320318 8422 raft_consensus.cc:2258] T 57015949a0a848b9a6ad41b12e1b876f P 73c558247bb64d0c9eed0af9f7b6d310 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:27:52.320618 8423 ts_tablet_manager.cc:1905] T 57015949a0a848b9a6ad41b12e1b876f P 14d8d8a3dc9e410eaaf247e5636ab762: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:52.320618 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 7960
I20260926 09:27:52.320935 8422 ts_tablet_manager.cc:1905] T 57015949a0a848b9a6ad41b12e1b876f P 73c558247bb64d0c9eed0af9f7b6d310: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:52.322706 8422 ts_tablet_manager.cc:1918] T 57015949a0a848b9a6ad41b12e1b876f P 73c558247bb64d0c9eed0af9f7b6d310: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:27:52.322762 8422 log.cc:1196] T 57015949a0a848b9a6ad41b12e1b876f P 73c558247bb64d0c9eed0af9f7b6d310: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/57015949a0a848b9a6ad41b12e1b876f
I20260926 09:27:52.322969 8422 ts_tablet_manager.cc:1939] T 57015949a0a848b9a6ad41b12e1b876f P 73c558247bb64d0c9eed0af9f7b6d310: Deleting consensus metadata
I20260926 09:27:52.323457 8424 ts_tablet_manager.cc:1905] T 57015949a0a848b9a6ad41b12e1b876f P 057eec7f37324574b965b69ae7eb1d27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:27:52.323503 7873 catalog_manager.cc:4841] TS 73c558247bb64d0c9eed0af9f7b6d310 (127.0.98.3:33271): tablet 57015949a0a848b9a6ad41b12e1b876f (table test-table [id=2944d4360747428e9f116af47549c3ae]) successfully deleted
I20260926 09:27:52.325201 8424 ts_tablet_manager.cc:1918] T 57015949a0a848b9a6ad41b12e1b876f P 057eec7f37324574b965b69ae7eb1d27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:27:52.325253 8424 log.cc:1196] T 57015949a0a848b9a6ad41b12e1b876f P 057eec7f37324574b965b69ae7eb1d27: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414839069479-392-0/minicluster-data/ts-1/wal/wals/57015949a0a848b9a6ad41b12e1b876f
I20260926 09:27:52.325474 8424 ts_tablet_manager.cc:1939] T 57015949a0a848b9a6ad41b12e1b876f P 057eec7f37324574b965b69ae7eb1d27: Deleting consensus metadata
I20260926 09:27:52.326066 7874 catalog_manager.cc:4841] TS 057eec7f37324574b965b69ae7eb1d27 (127.0.98.2:42267): tablet 57015949a0a848b9a6ad41b12e1b876f (table test-table [id=2944d4360747428e9f116af47549c3ae]) successfully deleted
W20260926 09:27:52.328473 7874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43171 (error 108)
I20260926 09:27:52.328631 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 8100
I20260926 09:27:52.330354 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:52.328894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43171 (local address 127.0.0.1:51824)
0926 09:27:52.330272 (+ 1378us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:52.330317 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1361,"thread_start_us":37,"threads_started":1}
W20260926 09:27:52.330475 7874 catalog_manager.cc:4584] TS 14d8d8a3dc9e410eaaf247e5636ab762 (127.0.98.1:43171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 57015949a0a848b9a6ad41b12e1b876f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43171: connect: Connection refused (error 111)
I20260926 09:27:52.334745 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 8240
I20260926 09:27:52.340200 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 7856
2026-09-26T09:27:52Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2704 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:27:52 dist-test-slave-dfzm krb5kdc[8490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:27:52 dist-test-slave-dfzm krb5kdc[8490](info): set up 2 sockets
Sep 26 09:27:52 dist-test-slave-dfzm krb5kdc[8490](info): commencing operation
krb5kdc: starting...
W20260926 09:27:53.374286 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:27:53 dist-test-slave-dfzm krb5kdc[8490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:27:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:27:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:27:53.494208 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34103
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:53.574329 8503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:53.574590 8503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:53.574630 8503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:53.575809 8503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:27:53.575879 8503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:53.575906 8503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:53.575961 8503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:27:53.575992 8503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:27:53.577557 8503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:53.577857 8503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:53.578092 8503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:53.582962 8508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:53.582994 8509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:53.583191 8503 server_base.cc:1034] running on GCE node
W20260926 09:27:53.582998 8511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:53.583501 8503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:53.583748 8503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:53.584911 8503 hybrid_clock.cc:648] HybridClock initialized: now 1790414873584894 us; error 48 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T09:27:53Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1250 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:27:53 dist-test-slave-dfzm krb5kdc[8516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:27:53 dist-test-slave-dfzm krb5kdc[8516](info): set up 2 sockets
Sep 26 09:27:53 dist-test-slave-dfzm krb5kdc[8516](info): commencing operation
krb5kdc: starting...
W20260926 09:27:54.662456 392 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:27:54 dist-test-slave-dfzm krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:27:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:27:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:27:54.780252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40641
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:54.859455 8529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:54.859719 8529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:54.859756 8529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:54.861027 8529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:27:54.861130 8529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:54.861160 8529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:54.861194 8529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:27:54.861217 8529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:27:54.862810 8529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:54.863098 8529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:54.863338 8529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:54.868099 8534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:54.868139 8537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:54.868153 8535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:54.868271 8529 server_base.cc:1034] running on GCE node
I20260926 09:27:54.868595 8529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:54.868845 8529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:54.869984 8529 hybrid_clock.cc:648] HybridClock initialized: now 1790414874869956 us; error 39 us; skew 500 ppm
Sep 26 09:27:54 dist-test-slave-dfzm krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:54.875727 8529 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:27:54.876156 8529 webserver.cc:460] Webserver started at http://127.0.98.62:36751/ using document root <none> and password file <none>
I20260926 09:27:54.876351 8529 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:54.876394 8529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:54.876487 8529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:54.877506 8529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "80b183b7e8764cc5bb1f0847210c3954"
format_stamp: "Formatted at 2026-09-26 09:27:54 on dist-test-slave-dfzm"
server_key: "40fd56dd9324a4c9d4601375c93f00d3"
server_key_iv: "dedac7ca1b5a2fb4b5b81e7513068c26"
server_key_version: "encryptionkey@0"
I20260926 09:27:54.877898 8529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "80b183b7e8764cc5bb1f0847210c3954"
format_stamp: "Formatted at 2026-09-26 09:27:54 on dist-test-slave-dfzm"
server_key: "40fd56dd9324a4c9d4601375c93f00d3"
server_key_iv: "dedac7ca1b5a2fb4b5b81e7513068c26"
server_key_version: "encryptionkey@0"
I20260926 09:27:54.879202 8529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:54.879918 8544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:54.880110 8529 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:54.880221 8529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "80b183b7e8764cc5bb1f0847210c3954"
format_stamp: "Formatted at 2026-09-26 09:27:54 on dist-test-slave-dfzm"
server_key: "40fd56dd9324a4c9d4601375c93f00d3"
server_key_iv: "dedac7ca1b5a2fb4b5b81e7513068c26"
server_key_version: "encryptionkey@0"
I20260926 09:27:54.880298 8529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:54.890535 8529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:54.892500 8529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:54.892666 8529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:54.896564 8529 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40641
I20260926 09:27:54.896615 8596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40641 every 8 connection(s)
I20260926 09:27:54.897145 8529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:27:54.897858 8597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:27:54.900092 8597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954: Bootstrap starting.
I20260926 09:27:54.900677 8597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954: Neither blocks nor log segments found. Creating new log.
I20260926 09:27:54.900970 8597 log.cc:824] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:54.901705 8597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954: No bootstrap required, opened a new log
I20260926 09:27:54.903262 8597 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80b183b7e8764cc5bb1f0847210c3954" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40641 } }
I20260926 09:27:54.903403 8597 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:27:54.903426 8597 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80b183b7e8764cc5bb1f0847210c3954, State: Initialized, Role: FOLLOWER
I20260926 09:27:54.903604 8597 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80b183b7e8764cc5bb1f0847210c3954" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40641 } }
I20260926 09:27:54.903668 8597 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:27:54.903693 8597 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:27:54.903765 8597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:27:54.904642 8597 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80b183b7e8764cc5bb1f0847210c3954" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40641 } }
I20260926 09:27:54.904757 8597 leader_election.cc:302] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80b183b7e8764cc5bb1f0847210c3954; no voters:
I20260926 09:27:54.904965 8597 leader_election.cc:288] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:27:54.905023 8600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:27:54.905088 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 8529
I20260926 09:27:54.905185 8600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [term 1 LEADER]: Becoming Leader. State: Replica: 80b183b7e8764cc5bb1f0847210c3954, State: Running, Role: LEADER
I20260926 09:27:54.905203 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:27:54.905257 8600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80b183b7e8764cc5bb1f0847210c3954" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40641 } }
I20260926 09:27:54.905392 392 external_mini_cluster.cc:1428] Setting key 6ad77cf7b90e8ee3fe4a395fe3152af9
I20260926 09:27:54.905263 8597 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:27:54.905570 8602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80b183b7e8764cc5bb1f0847210c3954. Latest consensus state: current_term: 1 leader_uuid: "80b183b7e8764cc5bb1f0847210c3954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80b183b7e8764cc5bb1f0847210c3954" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40641 } } }
I20260926 09:27:54.905553 8601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80b183b7e8764cc5bb1f0847210c3954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80b183b7e8764cc5bb1f0847210c3954" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40641 } } }
I20260926 09:27:54.905720 8602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:54.905751 8601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:54.906490 8604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:27:54.907040 8604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:27:54.908707 8604 catalog_manager.cc:1347] Generated new cluster ID: 2f155ce4d82c4a388f23ae8203952632
I20260926 09:27:54.908763 8604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:27:54 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:54.917313 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:54.906065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59806 (local address 127.0.98.62:40641)
0926 09:27:54.906213 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:54.906223 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:54.906256 (+ 33us) server_negotiation.cc:407] Connection header received
0926 09:27:54.906565 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:54.906572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:54.906669 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:54.907619 (+ 950us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:54.907711 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:54.909033 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:54.910154 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:54.910775 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:54.910871 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:54.913684 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:54.913690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:54.913690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:54.913704 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:54.915682 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:54.916057 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:54.916060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:54.916061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:54.916181 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:54.916384 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:54.916385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:54.916386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:54.916704 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:54.916862 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:54.916937 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 09:27:54.917080 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":66,"threads_started":1}
I20260926 09:27:54.923563 8604 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:27:54.924254 8604 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:27:54.933010 8604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954: Generated new TSK 0
I20260926 09:27:54.933213 8604 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:27:54.960525 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40641
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:55.044121 8623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:55.044370 8623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:55.044412 8623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:55.045719 8623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:55.045783 8623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:55.045831 8623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:27:55.047398 8623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:55.047668 8623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:55.047900 8623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:55.052959 8629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:55.052959 8628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:55.052994 8631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:55.053218 8623 server_base.cc:1034] running on GCE node
I20260926 09:27:55.053431 8623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:55.053727 8623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:55.054885 8623 hybrid_clock.cc:648] HybridClock initialized: now 1790414875054865 us; error 36 us; skew 500 ppm
Sep 26 09:27:55 dist-test-slave-dfzm krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:55.057478 8623 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:27:55.057991 8623 webserver.cc:460] Webserver started at http://127.0.98.1:36723/ using document root <none> and password file <none>
I20260926 09:27:55.058262 8623 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:55.058310 8623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:55.058475 8623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:55.059531 8623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "0a837f64b0ba4688ad89a7d29b6356e0"
format_stamp: "Formatted at 2026-09-26 09:27:55 on dist-test-slave-dfzm"
server_key: "f591e827e1a745faac5b343371205c86"
server_key_iv: "c8d17b0d04f9d86585bc1169ef512846"
server_key_version: "encryptionkey@0"
I20260926 09:27:55.059885 8623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0a837f64b0ba4688ad89a7d29b6356e0"
format_stamp: "Formatted at 2026-09-26 09:27:55 on dist-test-slave-dfzm"
server_key: "f591e827e1a745faac5b343371205c86"
server_key_iv: "c8d17b0d04f9d86585bc1169ef512846"
server_key_version: "encryptionkey@0"
I20260926 09:27:55.061226 8623 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:27:55.061985 8638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:55.062167 8623 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:55.062233 8623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "0a837f64b0ba4688ad89a7d29b6356e0"
format_stamp: "Formatted at 2026-09-26 09:27:55 on dist-test-slave-dfzm"
server_key: "f591e827e1a745faac5b343371205c86"
server_key_iv: "c8d17b0d04f9d86585bc1169ef512846"
server_key_version: "encryptionkey@0"
I20260926 09:27:55.062321 8623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:55.071444 8623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:55.073395 8623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:55.073565 8623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:55.073830 8623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:55.074159 8623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:55.074214 8623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:55.074254 8623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:55.074280 8623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:55.080304 8623 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45373
I20260926 09:27:55.080349 8751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45373 every 8 connection(s)
I20260926 09:27:55.080683 8623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:27:55.086165 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 8623
I20260926 09:27:55.086297 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:27:55.086529 392 external_mini_cluster.cc:1428] Setting key dfbbc20dcb8d6fd086711e195b0a76ac
Sep 26 09:27:55 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:55.091959 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:55.081518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49401 (local address 127.0.98.62:40641)
0926 09:27:55.081632 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:55.081634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:55.081835 (+ 201us) server_negotiation.cc:407] Connection header received
0926 09:27:55.082804 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:55.082806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:55.082816 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:55.082913 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:55.083622 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.084252 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.085265 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.085490 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.085545 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:55.088431 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:55.088433 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:55.088433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:55.088442 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:55.090370 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:55.090823 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:55.090825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:55.090825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:55.090894 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:55.091232 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:55.091233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:55.091234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:55.091367 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:55.091448 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:55.091751 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 09:27:55.091857 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:27:55.092159 8754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:55.081498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.1:49401)
0926 09:27:55.081690 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:55.081734 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:55.082686 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:55.082985 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:55.082990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:55.083300 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:55.083513 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:55.083517 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.084356 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.084357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:55.085187 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:55.085189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.085572 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.085574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:55.085766 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:55.086376 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:55.086387 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:55.088320 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:55.090472 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:55.090479 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:55.090481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:55.090738 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:55.090967 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:55.090968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:55.090968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:55.091108 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:55.091437 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:55.091443 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:55.091605 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 09:27:55.091727 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 09:27:55.091901 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":125,"threads_started":1}
I20260926 09:27:55.092706 8752 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40641
I20260926 09:27:55.092828 8752 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:55.093060 8752 heartbeater.cc:511] Master 127.0.98.62:40641 requested a full tablet report, sending...
I20260926 09:27:55.093631 8561 ts_manager.cc:194] Registered new tserver with Master: 0a837f64b0ba4688ad89a7d29b6356e0 (127.0.98.1:45373)
I20260926 09:27:55.094293 8561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49401
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:27:55.112872 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40641
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:55.193522 8757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:55.193801 8757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:55.193825 8757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:55.194938 8757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:55.194974 8757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:55.194999 8757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:27:55.196420 8757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:55.196601 8757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:55.196780 8757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:55.201675 8765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:55.201710 8762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:55.201761 8757 server_base.cc:1034] running on GCE node
W20260926 09:27:55.201706 8763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:55.202062 8757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:55.202271 8757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:55.203414 8757 hybrid_clock.cc:648] HybridClock initialized: now 1790414875203393 us; error 42 us; skew 500 ppm
Sep 26 09:27:55 dist-test-slave-dfzm krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:55.206137 8757 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:27:55.206626 8757 webserver.cc:460] Webserver started at http://127.0.98.2:33759/ using document root <none> and password file <none>
I20260926 09:27:55.206897 8757 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:55.206961 8757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:55.207093 8757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:55.208215 8757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "ab6838fccb814eceb13dde425f560cfa"
format_stamp: "Formatted at 2026-09-26 09:27:55 on dist-test-slave-dfzm"
server_key: "2950b97783258fb195909a6d732b7877"
server_key_iv: "b98a33dcd736e9f3d4426504b0f6abaa"
server_key_version: "encryptionkey@0"
I20260926 09:27:55.208575 8757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ab6838fccb814eceb13dde425f560cfa"
format_stamp: "Formatted at 2026-09-26 09:27:55 on dist-test-slave-dfzm"
server_key: "2950b97783258fb195909a6d732b7877"
server_key_iv: "b98a33dcd736e9f3d4426504b0f6abaa"
server_key_version: "encryptionkey@0"
I20260926 09:27:55.209947 8757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:27:55.210686 8772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:55.210844 8757 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:27:55.210940 8757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "ab6838fccb814eceb13dde425f560cfa"
format_stamp: "Formatted at 2026-09-26 09:27:55 on dist-test-slave-dfzm"
server_key: "2950b97783258fb195909a6d732b7877"
server_key_iv: "b98a33dcd736e9f3d4426504b0f6abaa"
server_key_version: "encryptionkey@0"
I20260926 09:27:55.211023 8757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:55.224987 8757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:55.227440 8757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:55.227619 8757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:55.227857 8757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:55.228158 8757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:55.228210 8757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:55.228250 8757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:55.228278 8757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:55.234225 8757 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42277
I20260926 09:27:55.234270 8885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42277 every 8 connection(s)
I20260926 09:27:55.234609 8757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:27:55.237534 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 8757
I20260926 09:27:55.237684 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:27:55.237901 392 external_mini_cluster.cc:1428] Setting key 037a935da90fa59bbfbab0475901525d
Sep 26 09:27:55 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:55.244912 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:55.235352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36495 (local address 127.0.98.62:40641)
0926 09:27:55.235435 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:55.235438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:55.235591 (+ 153us) server_negotiation.cc:407] Connection header received
0926 09:27:55.236521 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:55.236522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:55.236533 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:55.236595 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:55.237213 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.237853 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.239270 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.239456 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.239512 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:55.241858 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:55.241860 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:55.241860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:55.241868 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:55.243517 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:55.243926 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:55.243927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:55.243928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:55.243992 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:55.244229 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:55.244230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:55.244231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:55.244378 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:55.244457 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:55.244706 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 09:27:55.244804 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 09:27:55.244939 8888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:55.235304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.2:36495)
0926 09:27:55.235461 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:55.235498 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:55.236402 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:55.236688 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:55.236691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:55.236943 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:55.237135 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:55.237139 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.237940 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.237941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:55.239182 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:55.239185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.239537 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.239538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:55.239694 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:55.240334 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:55.240343 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:55.241764 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:55.243611 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:55.243624 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:55.243626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:55.243850 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:55.244065 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:55.244066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:55.244066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:55.244163 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:55.244450 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:55.244454 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:55.244566 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 09:27:55.244658 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 09:27:55.244776 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":94,"threads_started":1}
I20260926 09:27:55.245441 8886 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40641
I20260926 09:27:55.245541 8886 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:55.245771 8886 heartbeater.cc:511] Master 127.0.98.62:40641 requested a full tablet report, sending...
I20260926 09:27:55.246166 8561 ts_manager.cc:194] Registered new tserver with Master: ab6838fccb814eceb13dde425f560cfa (127.0.98.2:42277)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 09:27:55.246585 8561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:27:55.264801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40641
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:55.345388 8891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:55.345661 8891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:55.345723 8891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:55.346908 8891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:55.346985 8891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:55.347031 8891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:27:55.348851 8891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:55.349196 8891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:55.349445 8891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:55.354437 8896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:55.354432 8899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:55.354532 8897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:55.354679 8891 server_base.cc:1034] running on GCE node
I20260926 09:27:55.354904 8891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:55.355131 8891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:55.356276 8891 hybrid_clock.cc:648] HybridClock initialized: now 1790414875356266 us; error 37 us; skew 500 ppm
Sep 26 09:27:55 dist-test-slave-dfzm krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:55.358990 8891 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:27:55.359490 8891 webserver.cc:460] Webserver started at http://127.0.98.3:36091/ using document root <none> and password file <none>
I20260926 09:27:55.359731 8891 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:55.359776 8891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:55.359941 8891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:27:55.361001 8891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "eee1d872144f4be0b244fd596bfbc609"
format_stamp: "Formatted at 2026-09-26 09:27:55 on dist-test-slave-dfzm"
server_key: "797ec9b7729fefb650ff1d291b34256d"
server_key_iv: "539e4d1e67396c46f519147a35a762c1"
server_key_version: "encryptionkey@0"
I20260926 09:27:55.361370 8891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "eee1d872144f4be0b244fd596bfbc609"
format_stamp: "Formatted at 2026-09-26 09:27:55 on dist-test-slave-dfzm"
server_key: "797ec9b7729fefb650ff1d291b34256d"
server_key_iv: "539e4d1e67396c46f519147a35a762c1"
server_key_version: "encryptionkey@0"
I20260926 09:27:55.362843 8891 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:55.363561 8906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:55.363754 8891 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:55.363850 8891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "eee1d872144f4be0b244fd596bfbc609"
format_stamp: "Formatted at 2026-09-26 09:27:55 on dist-test-slave-dfzm"
server_key: "797ec9b7729fefb650ff1d291b34256d"
server_key_iv: "539e4d1e67396c46f519147a35a762c1"
server_key_version: "encryptionkey@0"
I20260926 09:27:55.363905 8891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:55.376910 8891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:55.379360 8891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:55.379480 8891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:55.379663 8891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:55.379974 8891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:55.380009 8891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:55.380064 8891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:55.380095 8891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:55.386001 8891 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34643
I20260926 09:27:55.386081 9019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34643 every 8 connection(s)
I20260926 09:27:55.386400 8891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:27:55.389624 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 8891
I20260926 09:27:55.389741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:27:55.389940 392 external_mini_cluster.cc:1428] Setting key 5354e39d58b5c59c7ad53703311e0f47
Sep 26 09:27:55 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:27:55.395926 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:55.387144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58279 (local address 127.0.98.62:40641)
0926 09:27:55.387213 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:55.387215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:55.387408 (+ 193us) server_negotiation.cc:407] Connection header received
0926 09:27:55.388382 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:55.388383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:55.388395 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:55.388457 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:55.389088 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.389789 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.391125 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.391314 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.391367 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:55.393471 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:55.393473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:55.393473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:55.393479 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:55.394795 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:55.395159 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:55.395160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:55.395161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:55.395212 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:55.395421 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:55.395421 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:55.395421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:55.395518 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:55.395567 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:55.395807 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 09:27:55.395864 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 09:27:55.396078 9022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:55.387143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.3:58279)
0926 09:27:55.387289 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:55.387323 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:55.388267 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:55.388531 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:55.388534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:55.388792 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:55.388981 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:55.388985 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.389876 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.389877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:55.391055 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:55.391057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.391384 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.391385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:55.391485 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:55.392039 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:55.392048 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:55.393390 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:27:55.394869 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:55.394875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:55.394876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:55.395085 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:55.395271 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:27:55.395271 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:27:55.395271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:27:55.395365 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:27:55.395572 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:27:55.395576 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:27:55.395696 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 09:27:55.395777 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 09:27:55.395891 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260926 09:27:55.396564 9020 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40641
I20260926 09:27:55.396641 9020 heartbeater.cc:464] Registering TS with master...
I20260926 09:27:55.396829 9020 heartbeater.cc:511] Master 127.0.98.62:40641 requested a full tablet report, sending...
I20260926 09:27:55.397197 8561 ts_manager.cc:194] Registered new tserver with Master: eee1d872144f4be0b244fd596bfbc609 (127.0.98.3:34643)
I20260926 09:27:55.397518 8561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58279
I20260926 09:27:55.401441 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 09:27:55.472440 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 8623
I20260926 09:27:55.477058 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 8757
I20260926 09:27:55.481365 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 8891
I20260926 09:27:55.485576 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 8529
I20260926 09:27:55.490458 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40641
--webserver_interface=127.0.98.62
--webserver_port=36751
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:27:55.570263 9029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:55.570485 9029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:55.570533 9029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:55.571741 9029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:27:55.571792 9029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:55.571807 9029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:55.571820 9029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:27:55.571832 9029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:27:55.573387 9029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:55.573663 9029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:55.573902 9029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:55.578694 9035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:55.578790 9037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:55.578790 9034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:55.579304 9029 server_base.cc:1034] running on GCE node
I20260926 09:27:55.579535 9029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:55.579844 9029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:55.581007 9029 hybrid_clock.cc:648] HybridClock initialized: now 1790414875580991 us; error 35 us; skew 500 ppm
Sep 26 09:27:55 dist-test-slave-dfzm krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:55.583488 9029 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:27:55.583959 9029 webserver.cc:460] Webserver started at http://127.0.98.62:36751/ using document root <none> and password file <none>
I20260926 09:27:55.584201 9029 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:55.584275 9029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:55.585660 9029 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:27:55.586331 9044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:55.586498 9029 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:27:55.586594 9029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "80b183b7e8764cc5bb1f0847210c3954"
format_stamp: "Formatted at 2026-09-26 09:27:54 on dist-test-slave-dfzm"
server_key: "40fd56dd9324a4c9d4601375c93f00d3"
server_key_iv: "dedac7ca1b5a2fb4b5b81e7513068c26"
server_key_version: "encryptionkey@0"
I20260926 09:27:55.586935 9029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:55.599220 9029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:55.600948 9029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:55.601107 9029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:55.605095 9029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40641
I20260926 09:27:55.605129 9096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40641 every 8 connection(s)
I20260926 09:27:55.605700 9029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:27:55.606935 9097 sys_catalog.cc:263] Verifying existing consensus state
I20260926 09:27:55.607561 9097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954: Bootstrap starting.
I20260926 09:27:55.615129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 9029
I20260926 09:27:55.615468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45373
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40641
--webserver_port=36723
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:27:55.615842 9097 log.cc:824] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954: Log is configured to *not* fsync() on all Append() calls
I20260926 09:27:55.617800 9097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:27:55.618111 9097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954: Bootstrap complete.
I20260926 09:27:55.620440 9097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80b183b7e8764cc5bb1f0847210c3954" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40641 } }
I20260926 09:27:55.620661 9097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80b183b7e8764cc5bb1f0847210c3954, State: Initialized, Role: FOLLOWER
I20260926 09:27:55.620839 9097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80b183b7e8764cc5bb1f0847210c3954" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40641 } }
I20260926 09:27:55.620910 9097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:27:55.620973 9097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:27:55.621012 9097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [term 1 FOLLOWER]: Advancing to term 2
I20260926 09:27:55.622133 9097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80b183b7e8764cc5bb1f0847210c3954" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40641 } }
I20260926 09:27:55.622224 9097 leader_election.cc:302] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80b183b7e8764cc5bb1f0847210c3954; no voters:
I20260926 09:27:55.622442 9097 leader_election.cc:288] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 09:27:55.622478 9101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 09:27:55.622591 9101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [term 2 LEADER]: Becoming Leader. State: Replica: 80b183b7e8764cc5bb1f0847210c3954, State: Running, Role: LEADER
I20260926 09:27:55.622680 9101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80b183b7e8764cc5bb1f0847210c3954" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40641 } }
I20260926 09:27:55.622728 9097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:27:55.622944 9101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "80b183b7e8764cc5bb1f0847210c3954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80b183b7e8764cc5bb1f0847210c3954" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40641 } } }
I20260926 09:27:55.623010 9101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:55.623118 9101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80b183b7e8764cc5bb1f0847210c3954. Latest consensus state: current_term: 2 leader_uuid: "80b183b7e8764cc5bb1f0847210c3954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80b183b7e8764cc5bb1f0847210c3954" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40641 } } }
I20260926 09:27:55.623183 9101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954 [sys.catalog]: This master's current role is: LEADER
I20260926 09:27:55.623616 9105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:27:55.624130 9105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:27:55.624297 9105 catalog_manager.cc:1259] Loaded cluster ID: 2f155ce4d82c4a388f23ae8203952632
I20260926 09:27:55.624533 9105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:27:55.625340 9105 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:27:55.625540 9105 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 80b183b7e8764cc5bb1f0847210c3954: Loaded TSK: 0
I20260926 09:27:55.625801 9105 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 09:27:55.716773 9100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:55.716980 9100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:55.717005 9100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:55.718201 9100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:55.718254 9100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:55.718281 9100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:27:55.719699 9100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36723
--tserver_master_addrs=127.0.98.62:40641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:55.719937 9100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:55.720202 9100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:55.726114 9123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:55.726146 9126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:55.726125 9124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:55.726609 9100 server_base.cc:1034] running on GCE node
I20260926 09:27:55.726845 9100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:55.727141 9100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:55.728333 9100 hybrid_clock.cc:648] HybridClock initialized: now 1790414875728312 us; error 33 us; skew 500 ppm
Sep 26 09:27:55 dist-test-slave-dfzm krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414875, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:55.731431 9100 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 09:27:55.731989 9100 webserver.cc:460] Webserver started at http://127.0.98.1:36723/ using document root <none> and password file <none>
I20260926 09:27:55.732237 9100 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:55.732326 9100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:55.733778 9100 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:27:55.734463 9133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:55.734661 9100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:27:55.734787 9100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "0a837f64b0ba4688ad89a7d29b6356e0"
format_stamp: "Formatted at 2026-09-26 09:27:55 on dist-test-slave-dfzm"
server_key: "f591e827e1a745faac5b343371205c86"
server_key_iv: "c8d17b0d04f9d86585bc1169ef512846"
server_key_version: "encryptionkey@0"
I20260926 09:27:55.735157 9100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:55.754671 9100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:55.757098 9100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:55.757270 9100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:55.757512 9100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:55.757905 9100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:55.757961 9100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:55.758006 9100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:55.758033 9100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:55.764673 9100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45373
I20260926 09:27:55.764719 9246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45373 every 8 connection(s)
I20260926 09:27:55.765065 9100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:27:55.770829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 9100
I20260926 09:27:55.771143 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42277
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40641
--webserver_port=33759
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:27:55 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:27:55 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:27:55.774928 9249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:55.765843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.1:47349)
0926 09:27:55.766030 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:55.766081 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:55.766543 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:55.767054 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:55.767058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:55.767615 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:55.767833 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:55.767837 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.769541 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.769542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:55.770405 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:55.770407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.771458 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.771460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:55.771627 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:55.772312 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:55.772323 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:55.774698 (+ 2375us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
W20260926 09:27:55.775008 9250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:55.765906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47349 (local address 127.0.98.62:40641)
0926 09:27:55.766126 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:55.766135 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:55.766183 (+ 48us) server_negotiation.cc:407] Connection header received
0926 09:27:55.766771 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:55.766781 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:55.766853 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:55.767019 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:55.767963 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.769429 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.770510 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.771118 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.771240 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:55.774839 (+ 3599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47349: BlockingRecv error: recv got EOF from 127.0.98.1:47349 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":91,"threads_started":1}
W20260926 09:27:55.775177 9247 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40641 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:40641: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 09:27:55 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:27:55 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:27:55.779975 9249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:55.775463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.1:37701)
0926 09:27:55.775687 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:55.775698 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:55.775776 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:55.775912 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:55.775913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:55.776049 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:55.776097 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:55.776098 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.776765 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.776766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:55.777283 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:55.777285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.777681 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.777681 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:55.777751 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:55.778191 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:55.778196 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:55.779884 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":83,"threads_started":1}
W20260926 09:27:55.780028 9250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:55.775656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37701 (local address 127.0.98.62:40641)
0926 09:27:55.775692 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:55.775694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:55.775760 (+ 66us) server_negotiation.cc:407] Connection header received
0926 09:27:55.775840 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:55.775841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:55.775856 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:55.775908 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:55.776175 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.776663 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.777379 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.777575 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.777682 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:55.779962 (+ 2280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37701: BlockingRecv error: recv got EOF from 127.0.98.1:37701 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 26 09:27:55 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:27:55 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:27:55.784691 9249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:55.780491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.1:55679)
0926 09:27:55.780577 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:55.780589 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:55.780637 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:55.780785 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:55.780785 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:55.780912 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:55.780940 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:55.780941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.781554 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.781556 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:55.782086 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:55.782088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.782449 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.782450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:55.782503 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:55.782982 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:55.782986 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:55.784607 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":69}
W20260926 09:27:55.784735 9250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:55.780535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55679 (local address 127.0.98.62:40641)
0926 09:27:55.780587 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:55.780589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:55.780622 (+ 33us) server_negotiation.cc:407] Connection header received
0926 09:27:55.780695 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:55.780696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:55.780706 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:55.780779 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:55.781020 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.781475 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.782170 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.782364 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.782449 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:55.784675 (+ 2226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55679: BlockingRecv error: recv got EOF from 127.0.98.1:55679 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 09:27:55.785120 9247 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 09:27:55.854334 9251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:55.854593 9251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:55.854637 9251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:55.855829 9251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:55.855901 9251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:55.855949 9251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:27:55.857462 9251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33759
--tserver_master_addrs=127.0.98.62:40641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:55.857789 9251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:55.858029 9251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:55.862924 9260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:55.862941 9257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:55.862937 9258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:55.863201 9251 server_base.cc:1034] running on GCE node
I20260926 09:27:55.863440 9251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:55.863677 9251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:55.864825 9251 hybrid_clock.cc:648] HybridClock initialized: now 1790414875864802 us; error 40 us; skew 500 ppm
Sep 26 09:27:55 dist-test-slave-dfzm krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414875, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:55.867480 9251 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 09:27:55.867959 9251 webserver.cc:460] Webserver started at http://127.0.98.2:33759/ using document root <none> and password file <none>
I20260926 09:27:55.868186 9251 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:55.868245 9251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:55.869542 9251 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:27:55.870160 9267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:55.870330 9251 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:55.870434 9251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "ab6838fccb814eceb13dde425f560cfa"
format_stamp: "Formatted at 2026-09-26 09:27:55 on dist-test-slave-dfzm"
server_key: "2950b97783258fb195909a6d732b7877"
server_key_iv: "b98a33dcd736e9f3d4426504b0f6abaa"
server_key_version: "encryptionkey@0"
I20260926 09:27:55.870787 9251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:55.882017 9251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:55.884162 9251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:55.884279 9251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:55.884459 9251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:55.884768 9251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:55.884807 9251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:55.884852 9251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:55.884878 9251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:55.890949 9251 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42277
I20260926 09:27:55.890987 9380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42277 every 8 connection(s)
I20260926 09:27:55.891331 9251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:27:55.896843 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 9251
I20260926 09:27:55.897157 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34643
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40641
--webserver_port=36091
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:27:55 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:27:55 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 09:27:55.899000 9250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:55.891996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52611 (local address 127.0.98.62:40641)
0926 09:27:55.892044 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:55.892047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:55.892315 (+ 268us) server_negotiation.cc:407] Connection header received
0926 09:27:55.892791 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:55.892792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:55.892805 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:55.892855 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:55.893847 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.894337 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.895332 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.895530 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.895598 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:55.898952 (+ 3354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52611: BlockingRecv error: recv got EOF from 127.0.98.2:52611 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:27:55.899101 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:55.892023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.2:52611)
0926 09:27:55.892183 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:55.892222 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:55.892685 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:55.892915 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:55.892918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:55.893477 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:55.893751 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:55.893756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.894422 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.894423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:55.895226 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:55.895228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.895614 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.895615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:55.895722 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:55.896206 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:55.896214 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:55.898864 (+ 2650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":100,"threads_started":1}
W20260926 09:27:55.899338 9381 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40641 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:40641: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 09:27:55 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:27:55 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:27:55.904245 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:55.899637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.2:42497)
0926 09:27:55.899693 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:55.899705 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:55.899752 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:55.899927 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:55.899928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:55.900069 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:55.900119 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:55.900120 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.900791 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.900792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:55.901333 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:55.901335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.901813 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.901814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:55.901886 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:55.902396 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:55.902402 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:55.904150 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 09:27:55.904302 9250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:55.899699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42497 (local address 127.0.98.62:40641)
0926 09:27:55.899749 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:55.899751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:55.899763 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:27:55.899822 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:55.899823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:55.899834 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:55.899922 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:55.900207 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.900690 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.901444 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.901722 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.901812 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:55.904234 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42497: BlockingRecv error: recv got EOF from 127.0.98.2:42497 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 26 09:27:55 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:27:55 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:27:55.909215 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:55.904811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.2:44425)
0926 09:27:55.904862 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:55.904873 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:55.904938 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:55.905102 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:55.905103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:55.905236 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:55.905264 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:55.905265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.906001 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.906002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:55.906501 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:55.906502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.906911 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.906912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:55.906964 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:55.907428 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:55.907432 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:55.909126 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 09:27:55.909271 9250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:55.904888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44425 (local address 127.0.98.62:40641)
0926 09:27:55.904948 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:55.904950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:55.904961 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:27:55.904993 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:55.904993 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:55.905004 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:55.905097 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:55.905346 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.905902 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.906615 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:55.906802 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:55.906897 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:55.909203 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44425: BlockingRecv error: recv got EOF from 127.0.98.2:44425 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 09:27:55.909636 9381 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 09:27:55.980463 9384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:27:55.980667 9384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:27:55.980691 9384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:27:55.981901 9384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:27:55.981963 9384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:27:55.981993 9384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:27:55.983402 9384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36091
--tserver_master_addrs=127.0.98.62:40641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:27:55.983592 9384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:27:55.983780 9384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:27:55.988752 9390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:55.988858 9384 server_base.cc:1034] running on GCE node
W20260926 09:27:55.988770 9389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:27:55.988744 9392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:27:55.989300 9384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:27:55.989624 9384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:27:55.990804 9384 hybrid_clock.cc:648] HybridClock initialized: now 1790414875990792 us; error 32 us; skew 500 ppm
Sep 26 09:27:55 dist-test-slave-dfzm krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414875, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:27:55.993352 9384 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 09:27:55.993804 9384 webserver.cc:460] Webserver started at http://127.0.98.3:36091/ using document root <none> and password file <none>
I20260926 09:27:55.993983 9384 fs_manager.cc:359] Metadata directory not provided
I20260926 09:27:55.994041 9384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:27:55.995318 9384 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:27:55.995844 9399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:55.995991 9384 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:27:55.996078 9384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "eee1d872144f4be0b244fd596bfbc609"
format_stamp: "Formatted at 2026-09-26 09:27:55 on dist-test-slave-dfzm"
server_key: "797ec9b7729fefb650ff1d291b34256d"
server_key_iv: "539e4d1e67396c46f519147a35a762c1"
server_key_version: "encryptionkey@0"
I20260926 09:27:55.996393 9384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:27:56.013247 9384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:27:56.015336 9384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:27:56.015458 9384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:27:56.015726 9384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:27:56.016005 9384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:27:56.016037 9384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:56.016060 9384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:27:56.016073 9384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:27:56.021934 9384 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34643
I20260926 09:27:56.021976 9512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34643 every 8 connection(s)
I20260926 09:27:56.022351 9384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:27:56.022636 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 9384
Sep 26 09:27:56 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:27:56 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 09:27:56.030418 9517 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:56.023495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35879 (local address 127.0.98.62:40641)
0926 09:27:56.023644 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:56.023647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:56.023781 (+ 134us) server_negotiation.cc:407] Connection header received
0926 09:27:56.024411 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:56.024413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:56.024425 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:56.024485 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:56.025456 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:56.026030 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:56.027040 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:56.027260 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:56.027346 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:56.030351 (+ 3005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35879: BlockingRecv error: recv got EOF from 127.0.98.3:35879 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":49,"threads_started":1}
I20260926 09:27:56.030490 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:56.023432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.3:35879)
0926 09:27:56.023652 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:56.023695 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:56.024302 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:56.024569 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:56.024573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:56.025146 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:56.025339 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:56.025343 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:56.026131 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:56.026133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:56.026935 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:56.026937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:56.027359 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:56.027360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:56.027491 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:56.028109 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:56.028117 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:56.030284 (+ 2167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":103,"threads_started":1}
I20260926 09:27:56.030660 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:56.023106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59820 (local address 127.0.98.62:40641)
0926 09:27:56.023177 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:56.023180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:56.023395 (+ 215us) server_negotiation.cc:407] Connection header received
0926 09:27:56.023478 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:56.023480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:56.023494 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:56.023565 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:56.024347 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:56.024838 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:56.025639 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:56.025868 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:56.025958 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:56.027273 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:27:56.027278 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:27:56.027278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:27:56.027291 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:27:56.029171 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:56.029538 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:56.029541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:56.029542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:56.029691 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:27:56.029912 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:27:56.029913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:27:56.029913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:27:56.030149 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:27:56.030309 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 09:27:56.030422 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 09:27:56.030550 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 09:27:56.030690 9513 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40641 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:40641: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 09:27:56 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:27:56 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:27:56.035286 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:56.030908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.3:38269)
0926 09:27:56.030967 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:56.031034 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:56.031095 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:56.031229 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:56.031230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:56.031351 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:56.031422 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:56.031424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:56.032149 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:56.032150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:56.032649 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:56.032650 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:56.032953 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:56.032954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:56.033003 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:56.033423 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:56.033427 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:56.035200 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 09:27:56.035351 9250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:56.031002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38269 (local address 127.0.98.62:40641)
0926 09:27:56.031034 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:56.031036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:56.031079 (+ 43us) server_negotiation.cc:407] Connection header received
0926 09:27:56.031169 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:56.031170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:56.031179 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:56.031227 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:56.031541 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:56.032064 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:56.032720 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:56.032891 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:56.032939 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:56.035289 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38269: BlockingRecv error: recv got EOF from 127.0.98.3:38269 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Sep 26 09:27:56 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:27:56 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:27:56.039994 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:56.035639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.3:57643)
0926 09:27:56.035688 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:56.035699 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:56.035737 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:56.035920 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:56.035921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:56.036047 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:56.036072 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:56.036073 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:56.036680 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:56.036681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:56.037173 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:56.037175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:56.037570 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:56.037571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:56.037643 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:56.038072 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:56.038077 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:56.039909 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 09:27:56.040040 9250 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:56.035738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57643 (local address 127.0.98.62:40641)
0926 09:27:56.035802 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:56.035804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:56.035812 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:27:56.035831 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:56.035832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:56.035840 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:56.035915 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:56.036151 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:56.036602 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:56.037276 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:56.037485 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:56.037567 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:56.039977 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57643: BlockingRecv error: recv got EOF from 127.0.98.3:57643 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 09:27:56.040159 9513 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 09:27:56 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:27:56 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:27:56.790334 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:56.785574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.1:44853)
0926 09:27:56.785761 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:56.785772 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:56.785831 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:56.786005 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:56.786006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:56.786161 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:56.786232 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:56.786233 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:56.786880 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:56.786881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:56.787418 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:56.787420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:56.787727 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:56.787728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:56.787774 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:56.788274 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:56.788278 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:56.790217 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":72,"threads_started":1}
W20260926 09:27:56.790449 9519 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:56.785733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44853 (local address 127.0.98.62:40641)
0926 09:27:56.785853 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:56.785856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:56.785876 (+ 20us) server_negotiation.cc:407] Connection header received
0926 09:27:56.785904 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:56.785905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:56.785916 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:56.785990 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:56.786319 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:56.786807 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:56.787487 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:56.787665 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:56.787714 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:56.790317 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44853: BlockingRecv error: recv got EOF from 127.0.98.1:44853 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":52,"threads_started":1}
Sep 26 09:27:56 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:27:56 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:27:56.914939 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:56.910110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.2:52859)
0926 09:27:56.910240 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:56.910253 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:56.910341 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:56.910522 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:56.910523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:56.910727 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:56.910793 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:56.910795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:56.911430 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:56.911430 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:56.912072 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:56.912074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:56.912387 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:56.912388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:56.912434 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:56.912963 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:56.912968 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:56.914875 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
W20260926 09:27:56.915043 9519 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:56.910156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52859 (local address 127.0.98.62:40641)
0926 09:27:56.910203 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:56.910205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:56.910323 (+ 118us) server_negotiation.cc:407] Connection header received
0926 09:27:56.910418 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:56.910419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:56.910432 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:56.910499 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:56.910861 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:56.911340 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:56.912145 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:56.912324 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:56.912374 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:56.914968 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52859: BlockingRecv error: recv got EOF from 127.0.98.2:52859 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 26 09:27:57 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:27:57 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:27:57.045197 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:57.040608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.3:53359)
0926 09:27:57.040788 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:57.040802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:57.040868 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:57.041030 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:57.041031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:57.041261 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:57.041321 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:57.041323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:57.041974 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:57.041975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:57.042525 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:57.042526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:57.042838 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:57.042839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:57.042878 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:57.043394 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:57.043398 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:57.045129 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
W20260926 09:27:57.045383 9519 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:57.040696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53359 (local address 127.0.98.62:40641)
0926 09:27:57.040745 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:57.040747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:57.040876 (+ 129us) server_negotiation.cc:407] Connection header received
0926 09:27:57.040932 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:57.040933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:57.040947 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:57.041019 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:57.041408 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:57.041906 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:57.042603 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:57.042776 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:57.042827 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:57.045219 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53359: BlockingRecv error: recv got EOF from 127.0.98.3:53359 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 26 09:27:57 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:27:57 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:27:57.795845 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:57.790907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.1:33851)
0926 09:27:57.791079 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:57.791090 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:57.791150 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:57.791328 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:57.791329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:57.791549 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:57.791609 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:57.791610 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:57.792264 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:57.792264 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:57.792778 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:57.792779 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:57.793107 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:57.793108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:57.793146 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:57.793753 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:57.793759 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:57.795745 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":113,"threads_started":1}
W20260926 09:27:57.795900 9523 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:57.791004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33851 (local address 127.0.98.62:40641)
0926 09:27:57.791100 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:57.791104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:57.791141 (+ 37us) server_negotiation.cc:407] Connection header received
0926 09:27:57.791226 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:57.791227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:57.791237 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:57.791301 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:57.791692 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:57.792194 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:57.792848 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:57.793042 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:57.793093 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:57.795825 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33851: BlockingRecv error: recv got EOF from 127.0.98.1:33851 (error 108)
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":47,"threads_started":1}
Sep 26 09:27:57 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:27:57 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:27:57.919862 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:57.915468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.2:56623)
0926 09:27:57.915641 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:57.915651 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:57.915709 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:57.915859 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:57.915860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:57.916017 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:57.916064 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:57.916065 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:57.916733 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:57.916734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:57.917275 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:57.917277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:57.917653 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:57.917654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:57.917704 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:57.918135 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:57.918139 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:57.919803 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":112,"threads_started":1}
W20260926 09:27:57.920004 9523 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:57.915519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56623 (local address 127.0.98.62:40641)
0926 09:27:57.915569 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:57.915571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:57.915713 (+ 142us) server_negotiation.cc:407] Connection header received
0926 09:27:57.915749 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:57.915750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:57.915760 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:57.915811 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:57.916150 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:57.916632 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:57.917353 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:57.917531 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:57.917630 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:57.919917 (+ 2287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56623: BlockingRecv error: recv got EOF from 127.0.98.2:56623 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 26 09:27:58 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:27:58 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:27:58.050347 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:58.045696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.3:50953)
0926 09:27:58.045845 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:58.045862 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:58.045909 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:58.046098 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:58.046099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:58.046261 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:58.046310 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:58.046311 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:58.046987 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:58.046988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:58.047524 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:58.047525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:58.047852 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:58.047853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:58.047896 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:58.048360 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:58.048365 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:58.050275 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
W20260926 09:27:58.050505 9523 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:58.045867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50953 (local address 127.0.98.62:40641)
0926 09:27:58.045927 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:58.045929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:58.045938 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:27:58.045968 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:58.045968 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:58.045981 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:58.046070 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:58.046415 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:58.046904 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:58.047601 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:58.047781 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:58.047841 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:58.050418 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50953: BlockingRecv error: recv got EOF from 127.0.98.3:50953 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 26 09:27:58 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:27:58 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:27:58.800961 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:58.796355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.1:35429)
0926 09:27:58.796510 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:58.796521 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:58.796588 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:58.796722 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:58.796722 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:58.796902 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:58.796953 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:58.796955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:58.797664 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:58.797665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:58.798192 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:58.798193 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:58.798571 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:58.798572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:58.798617 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:58.799050 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:58.799055 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:58.800903 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
W20260926 09:27:58.801098 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:58.796450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35429 (local address 127.0.98.62:40641)
0926 09:27:58.796572 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:58.796574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:58.796607 (+ 33us) server_negotiation.cc:407] Connection header received
0926 09:27:58.796634 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:58.796634 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:58.796644 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:58.796704 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:58.797021 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:58.797528 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:58.798264 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:58.798500 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:58.798553 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:58.801012 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35429: BlockingRecv error: recv got EOF from 127.0.98.1:35429 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":52,"threads_started":1}
Sep 26 09:27:58 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:27:58 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:27:58.925019 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:58.920330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.2:55193)
0926 09:27:58.920453 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:58.920464 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:58.920514 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:58.920683 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:58.920684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:58.920845 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:58.920894 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:58.920895 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:58.921631 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:58.921632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:58.922228 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:58.922230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:58.922544 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:58.922545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:58.922605 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:58.923153 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:58.923157 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:58.924961 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":67,"threads_started":1}
W20260926 09:27:58.925154 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:58.920413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55193 (local address 127.0.98.62:40641)
0926 09:27:58.920503 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:58.920505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:58.920519 (+ 14us) server_negotiation.cc:407] Connection header received
0926 09:27:58.920580 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:58.920581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:58.920592 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:58.920647 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:58.920991 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:58.921489 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:58.922295 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:58.922480 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:58.922535 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:58.925072 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55193: BlockingRecv error: recv got EOF from 127.0.98.2:55193 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 09:27:59 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:27:59 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:27:59.055389 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:59.050797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.3:49243)
0926 09:27:59.050949 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:59.050970 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:59.051031 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:59.051200 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:59.051202 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:59.051367 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:59.051415 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:59.051417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:59.052054 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:59.052055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:59.052572 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:59.052574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:59.052896 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:59.052896 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:59.052939 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:59.053468 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:59.053473 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:59.055323 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
W20260926 09:27:59.055464 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:59.050859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49243 (local address 127.0.98.62:40641)
0926 09:27:59.050904 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:59.050906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:59.051031 (+ 125us) server_negotiation.cc:407] Connection header received
0926 09:27:59.051087 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:59.051088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:59.051100 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:59.051152 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:59.051489 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:59.051983 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:59.052650 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:59.052825 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:59.052871 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:59.055414 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49243: BlockingRecv error: recv got EOF from 127.0.98.3:49243 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 09:27:59 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:27:59 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:27:59.805761 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:59.801420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.1:58945)
0926 09:27:59.801557 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:59.801571 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:59.801678 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:59.801830 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:59.801831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:59.801987 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:59.802037 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:59.802038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:59.802656 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:59.802657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:59.803177 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:59.803178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:59.803491 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:59.803492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:59.803529 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:59.803966 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:59.803971 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:59.805669 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
W20260926 09:27:59.805840 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:59.801463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58945 (local address 127.0.98.62:40641)
0926 09:27:59.801604 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:59.801606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:59.801661 (+ 55us) server_negotiation.cc:407] Connection header received
0926 09:27:59.801746 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:59.801747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:59.801757 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:59.801808 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:59.802106 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:59.802590 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:59.803246 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:59.803428 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:59.803479 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:59.805766 (+ 2287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58945: BlockingRecv error: recv got EOF from 127.0.98.1:58945 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":78,"threads_started":1}
Sep 26 09:27:59 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:27:59 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:27:59.933885 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:27:59.925630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.2:59885)
0926 09:27:59.925789 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 09:27:59.925799 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:27:59.925856 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:27:59.925984 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:27:59.925985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:27:59.926139 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:27:59.926209 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:59.926211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:59.926866 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:59.926867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:59.927389 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:27:59.927390 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:27:59.927725 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:27:59.927725 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:27:59.927762 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:59.932144 (+ 4382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:27:59.932149 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:27:59.933829 (+ 1680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
W20260926 09:27:59.933990 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:27:59.925672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59885 (local address 127.0.98.62:40641)
0926 09:27:59.925756 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 09:27:59.925758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:27:59.925859 (+ 101us) server_negotiation.cc:407] Connection header received
0926 09:27:59.925900 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:27:59.925901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:27:59.925911 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:27:59.925966 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:27:59.926292 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:59.926798 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:59.927474 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:27:59.927659 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:27:59.927714 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:27:59.933921 (+ 6207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59885: BlockingRecv error: recv got EOF from 127.0.98.2:59885 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 09:28:00 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:00 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:00.060503 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:00.055980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.3:46237)
0926 09:28:00.056152 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:00.056164 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:00.056230 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:00.056384 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:00.056385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:00.056543 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:00.056593 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:00.056595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:00.057223 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:00.057223 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:00.057783 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:00.057786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:00.058107 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:00.058107 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:00.058147 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:00.058627 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:00.058632 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:00.060375 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":93,"threads_started":1}
W20260926 09:28:00.060524 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:00.056055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46237 (local address 127.0.98.62:40641)
0926 09:28:00.056150 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:00.056153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:00.056216 (+ 63us) server_negotiation.cc:407] Connection header received
0926 09:28:00.056288 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:00.056289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:00.056301 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:00.056352 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:00.056660 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:00.057145 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:00.057859 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:00.058041 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:00.058097 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:00.060473 (+ 2376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46237: BlockingRecv error: recv got EOF from 127.0.98.3:46237 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 09:28:00 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:00 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:00.810848 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:00.806255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.1:53701)
0926 09:28:00.806409 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:00.806429 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:00.806479 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:00.806631 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:00.806632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:00.806789 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:00.806836 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:00.806837 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:00.807487 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:00.807488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:00.808009 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:00.808010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:00.808349 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:00.808350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:00.808389 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:00.808851 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:00.808856 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:00.810734 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":113,"threads_started":1}
W20260926 09:28:00.810891 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:00.806336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53701 (local address 127.0.98.62:40641)
0926 09:28:00.806469 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:00.806471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:00.806482 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:28:00.806550 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:00.806551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:00.806561 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:00.806613 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:00.806910 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:00.807420 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:00.808077 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:00.808258 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:00.808321 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:00.810815 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53701: BlockingRecv error: recv got EOF from 127.0.98.1:53701 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
Sep 26 09:28:00 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:00 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:00.939258 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:00.934420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.2:36549)
0926 09:28:00.934546 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:00.934562 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:00.934679 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:00.934864 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:00.934865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:00.935045 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:00.935091 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:00.935092 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:00.935759 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:00.935761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:00.936292 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:00.936293 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:00.936680 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:00.936680 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:00.936742 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:00.937227 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:00.937231 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:00.939187 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
W20260926 09:28:00.939357 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:00.934560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36549 (local address 127.0.98.62:40641)
0926 09:28:00.934641 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:00.934643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:00.934653 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:28:00.934786 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:00.934787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:00.934798 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:00.934857 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:00.935179 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:00.935671 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:00.936375 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:00.936567 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:00.936666 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:00.939284 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36549: BlockingRecv error: recv got EOF from 127.0.98.2:36549 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 09:28:01 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:01 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:01.065626 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:01.061096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.3:50013)
0926 09:28:01.061257 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:01.061268 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:01.061325 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:01.061465 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:01.061465 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:01.061651 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:01.061698 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:01.061700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:01.062326 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:01.062327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:01.062898 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:01.062901 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:01.063259 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:01.063260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:01.063305 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:01.063766 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:01.063771 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:01.065513 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
W20260926 09:28:01.065686 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:01.061212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50013 (local address 127.0.98.62:40641)
0926 09:28:01.061259 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:01.061261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:01.061312 (+ 51us) server_negotiation.cc:407] Connection header received
0926 09:28:01.061389 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:01.061389 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:01.061401 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:01.061442 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:01.061755 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:01.062249 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:01.062983 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:01.063182 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:01.063240 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:01.065637 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50013: BlockingRecv error: recv got EOF from 127.0.98.3:50013 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 09:28:01 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:01 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:01.816431 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:01.811467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.1:58581)
0926 09:28:01.811587 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:01.811605 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:01.811653 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:01.811847 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:01.811848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:01.812009 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:01.812058 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:01.812059 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:01.812729 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:01.812730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:01.813431 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:01.813433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:01.813829 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:01.813830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:01.813870 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:01.814399 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:01.814410 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:01.816310 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
W20260926 09:28:01.816468 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:01.811550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58581 (local address 127.0.98.62:40641)
0926 09:28:01.811702 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:01.811706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:01.811719 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:28:01.811745 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:01.811745 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:01.811758 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:01.811832 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:01.812145 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:01.812647 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:01.813529 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:01.813764 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:01.813824 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:01.816394 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58581: BlockingRecv error: recv got EOF from 127.0.98.1:58581 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
Sep 26 09:28:01 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:01 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:01.944443 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:01.939797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.2:53217)
0926 09:28:01.939951 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:01.939963 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:01.940034 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:01.940165 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:01.940165 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:01.940333 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:01.940380 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:01.940382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:01.941015 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:01.941016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:01.941551 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:01.941553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:01.941924 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:01.941924 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:01.941973 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:01.942495 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:01.942499 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:01.944335 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":69,"threads_started":1}
W20260926 09:28:01.944605 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:01.939864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53217 (local address 127.0.98.62:40641)
0926 09:28:01.939949 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:01.939951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:01.940017 (+ 66us) server_negotiation.cc:407] Connection header received
0926 09:28:01.940085 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:01.940086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:01.940096 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:01.940146 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:01.940447 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:01.940927 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:01.941662 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:01.941843 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:01.941893 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:01.944533 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53217: BlockingRecv error: recv got EOF from 127.0.98.2:53217 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 09:28:02 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:02 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:02.070766 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:02.066142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.3:34749)
0926 09:28:02.066275 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:02.066287 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:02.066351 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:02.066505 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:02.066505 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:02.066654 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:02.066701 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:02.066702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:02.067399 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:02.067400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:02.067950 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:02.067952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:02.068308 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:02.068309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:02.068349 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:02.068825 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:02.068829 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:02.070707 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":65,"threads_started":1}
W20260926 09:28:02.070968 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:02.066192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34749 (local address 127.0.98.62:40641)
0926 09:28:02.066243 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:02.066246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:02.066340 (+ 94us) server_negotiation.cc:407] Connection header received
0926 09:28:02.066413 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:02.066413 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:02.066432 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:02.066486 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:02.066769 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:02.067321 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:02.068036 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:02.068237 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:02.068295 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:02.070870 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34749: BlockingRecv error: recv got EOF from 127.0.98.3:34749 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 26 09:28:02 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:02 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:02.821727 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:02.816970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.1:45745)
0926 09:28:02.817164 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:02.817178 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:02.817236 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:02.817478 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:02.817479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:02.817760 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:02.817825 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:02.817827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:02.818579 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:02.818580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:02.819092 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:02.819093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:02.819475 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:02.819476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:02.819514 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:02.820005 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:02.820009 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:02.821669 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":101,"threads_started":1}
W20260926 09:28:02.821831 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:02.817095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45745 (local address 127.0.98.62:40641)
0926 09:28:02.817207 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:02.817231 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:02.817241 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:28:02.817298 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:02.817298 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:02.817310 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:02.817385 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:02.817920 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:02.818490 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:02.819176 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:02.819406 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:02.819468 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:02.821756 (+ 2288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45745: BlockingRecv error: recv got EOF from 127.0.98.1:45745 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":57,"threads_started":1}
Sep 26 09:28:02 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:02 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:02.949930 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:02.945013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.2:57187)
0926 09:28:02.945166 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:02.945177 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:02.945230 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:02.945435 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:02.945435 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:02.945676 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:02.945724 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:02.945725 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:02.946374 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:02.946374 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:02.946976 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:02.946978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:02.947464 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:02.947465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:02.947532 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:02.948126 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:02.948130 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:02.949859 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
W20260926 09:28:02.950038 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:02.945163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57187 (local address 127.0.98.62:40641)
0926 09:28:02.945270 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:02.945271 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:02.945281 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:28:02.945310 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:02.945311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:02.945321 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:02.945382 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:02.945803 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:02.946292 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:02.947065 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:02.947377 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:02.947447 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:02.949946 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57187: BlockingRecv error: recv got EOF from 127.0.98.2:57187 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 09:28:03 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:03 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:03.075903 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:03.071341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.3:47623)
0926 09:28:03.071494 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:03.071511 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:03.071572 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:03.071716 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:03.071716 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:03.071869 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:03.071916 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:03.071918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:03.072541 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:03.072542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:03.073052 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:03.073053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:03.073362 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:03.073363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:03.073401 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:03.073972 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:03.073977 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:03.075843 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
W20260926 09:28:03.076030 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:03.071432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47623 (local address 127.0.98.62:40641)
0926 09:28:03.071491 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:03.071494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:03.071576 (+ 82us) server_negotiation.cc:407] Connection header received
0926 09:28:03.071629 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:03.071630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:03.071641 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:03.071698 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:03.071971 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:03.072462 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:03.073130 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:03.073303 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:03.073350 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:03.075944 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47623: BlockingRecv error: recv got EOF from 127.0.98.3:47623 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 26 09:28:03 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:03 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:03.826942 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:03.822299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.1:44499)
0926 09:28:03.822431 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:03.822443 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:03.822499 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:03.822718 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:03.822720 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:03.822879 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:03.822927 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:03.822928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:03.823674 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:03.823676 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:03.824332 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:03.824334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:03.824658 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:03.824659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:03.824696 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:03.825126 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:03.825130 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:03.826885 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
W20260926 09:28:03.827090 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:03.822394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44499 (local address 127.0.98.62:40641)
0926 09:28:03.822510 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:03.822513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:03.822568 (+ 55us) server_negotiation.cc:407] Connection header received
0926 09:28:03.822593 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:03.822594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:03.822606 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:03.822703 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:03.823021 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:03.823566 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:03.824405 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:03.824591 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:03.824644 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:03.826999 (+ 2355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44499: BlockingRecv error: recv got EOF from 127.0.98.1:44499 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":68,"threads_started":1}
Sep 26 09:28:03 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:03 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:03.954955 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:03.950436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.2:53963)
0926 09:28:03.950586 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:03.950598 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:03.950671 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:03.950840 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:03.950840 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:03.951046 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:03.951099 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:03.951101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:03.951857 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:03.951858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:03.952404 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:03.952406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:03.952754 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:03.952755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:03.952803 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:03.953238 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:03.953242 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:03.954898 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
W20260926 09:28:03.955084 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:03.950504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53963 (local address 127.0.98.62:40641)
0926 09:28:03.950573 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:03.950575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:03.950685 (+ 110us) server_negotiation.cc:407] Connection header received
0926 09:28:03.950738 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:03.950739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:03.950752 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:03.950831 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:03.951165 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:03.951789 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:03.952478 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:03.952669 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:03.952739 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:03.955003 (+ 2264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53963: BlockingRecv error: recv got EOF from 127.0.98.2:53963 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 09:28:04 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:04 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:04.081010 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:04.076432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.3:47875)
0926 09:28:04.076583 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:04.076594 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:04.076667 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:04.076804 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:04.076804 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:04.076964 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:04.077009 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:04.077010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:04.077668 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:04.077669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:04.078191 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:04.078192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:04.078508 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:04.078509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:04.078547 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:04.079068 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:04.079073 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:04.080908 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
W20260926 09:28:04.081067 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:04.076497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47875 (local address 127.0.98.62:40641)
0926 09:28:04.076581 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:04.076583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:04.076663 (+ 80us) server_negotiation.cc:407] Connection header received
0926 09:28:04.076726 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:04.076727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:04.076740 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:04.076782 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:04.077066 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:04.077573 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:04.078260 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:04.078441 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:04.078496 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:04.080996 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47875: BlockingRecv error: recv got EOF from 127.0.98.3:47875 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 09:28:04 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:04 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:04.832512 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:04.827496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.1:54275)
0926 09:28:04.827641 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:04.827651 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:04.827702 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:04.827879 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:04.827879 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:04.828035 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:04.828084 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:04.828085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:04.828702 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:04.828702 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:04.829263 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:04.829265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:04.829796 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:04.829798 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:04.829873 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:04.830477 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:04.830482 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:04.832452 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
W20260926 09:28:04.832641 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:04.827559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54275 (local address 127.0.98.62:40641)
0926 09:28:04.827696 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:04.827698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:04.827708 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:28:04.827772 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:04.827773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:04.827783 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:04.827863 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:04.828150 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:04.828636 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:04.829392 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:04.829672 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:04.829759 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:04.832581 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54275: BlockingRecv error: recv got EOF from 127.0.98.1:54275 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
Sep 26 09:28:04 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:04 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:04.964893 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:04.955502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.2:55561)
0926 09:28:04.955651 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:04.955663 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:04.955724 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:04.955876 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:04.955877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:04.956038 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:04.956084 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:04.956086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:04.956721 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:04.956722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:04.957238 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:04.957239 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:04.957622 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:04.957623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:04.957663 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:04.962888 (+ 5225us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:04.962894 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:04.964794 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
W20260926 09:28:04.964950 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:04.955642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55561 (local address 127.0.98.62:40641)
0926 09:28:04.955707 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:04.955709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:04.955720 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:28:04.955784 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:04.955784 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:04.955795 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:04.955859 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:04.956162 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:04.956647 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:04.957330 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:04.957520 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:04.957677 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:04.964886 (+ 7209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55561: BlockingRecv error: recv got EOF from 127.0.98.2:55561 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 09:28:05 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:05 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:05.086992 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:05.081749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.3:57033)
0926 09:28:05.081906 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:05.081918 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:05.081983 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:05.082144 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:05.082145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:05.082413 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:05.082461 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:05.082463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:05.083266 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:05.083267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:05.083813 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:05.083814 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:05.084199 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:05.084200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:05.084246 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:05.084781 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:05.084786 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:05.086932 (+ 2146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":88,"threads_started":1}
W20260926 09:28:05.087153 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:05.081848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57033 (local address 127.0.98.62:40641)
0926 09:28:05.081912 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:05.081914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:05.081965 (+ 51us) server_negotiation.cc:407] Connection header received
0926 09:28:05.082040 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:05.082041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:05.082057 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:05.082124 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:05.082531 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:05.083110 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:05.083897 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:05.084099 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:05.084163 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:05.087045 (+ 2882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57033: BlockingRecv error: recv got EOF from 127.0.98.3:57033 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 09:28:05 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:05 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:05.837626 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:05.833193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.1:53943)
0926 09:28:05.833327 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:05.833342 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:05.833393 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:05.833560 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:05.833561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:05.833741 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:05.833789 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:05.833791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:05.834400 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:05.834400 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:05.834923 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:05.834924 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:05.835242 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:05.835243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:05.835288 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:05.835745 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:05.835750 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:05.837508 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
W20260926 09:28:05.837690 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:05.833263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53943 (local address 127.0.98.62:40641)
0926 09:28:05.833418 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:05.833440 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:05.833448 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:28:05.833471 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:05.833471 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:05.833481 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:05.833543 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:05.833853 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:05.834333 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:05.834994 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:05.835177 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:05.835226 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:05.837615 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53943: BlockingRecv error: recv got EOF from 127.0.98.1:53943 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
Sep 26 09:28:05 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:05 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:05.970319 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:05.965461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.2:34285)
0926 09:28:05.965655 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:05.965668 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:05.965761 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:05.965942 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:05.965943 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:05.966113 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:05.966157 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:05.966159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:05.966928 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:05.966930 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:05.967533 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:05.967534 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:05.967929 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:05.967930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:05.968001 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:05.968506 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:05.968518 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:05.970222 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":93,"threads_started":1}
W20260926 09:28:05.970363 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:05.965572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34285 (local address 127.0.98.62:40641)
0926 09:28:05.965659 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:05.965661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:05.965742 (+ 81us) server_negotiation.cc:407] Connection header received
0926 09:28:05.965853 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:05.965854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:05.965866 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:05.965920 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:05.966239 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:05.966864 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:05.967626 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:05.967840 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:05.967929 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:05.970300 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34285: BlockingRecv error: recv got EOF from 127.0.98.2:34285 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 09:28:06 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:06 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:06.092568 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:06.087584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.3:60707)
0926 09:28:06.087755 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:06.087767 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:06.087876 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:06.088049 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:06.088050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:06.088234 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:06.088278 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:06.088280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:06.088962 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:06.088963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:06.089517 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:06.089519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:06.089946 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:06.089947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:06.089998 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:06.090640 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:06.090644 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:06.092466 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
W20260926 09:28:06.092622 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:06.087789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60707 (local address 127.0.98.62:40641)
0926 09:28:06.087857 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:06.087860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:06.087870 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:28:06.087940 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:06.087941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:06.087976 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:06.088043 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:06.088364 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:06.088866 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:06.089626 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:06.089871 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:06.089948 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:06.092557 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60707: BlockingRecv error: recv got EOF from 127.0.98.3:60707 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 09:28:06 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:06 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:06.843041 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:06.838101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.1:57471)
0926 09:28:06.838251 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:06.838266 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:06.838331 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:06.838515 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:06.838515 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:06.838726 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:06.838775 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:06.838777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:06.839515 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:06.839517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:06.840046 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:06.840048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:06.840434 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:06.840434 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:06.840475 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:06.840935 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:06.840940 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:06.842978 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
W20260926 09:28:06.843174 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:06.838173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57471 (local address 127.0.98.62:40641)
0926 09:28:06.838344 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:06.838347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:06.838382 (+ 35us) server_negotiation.cc:407] Connection header received
0926 09:28:06.838429 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:06.838430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:06.838441 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:06.838496 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:06.838843 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:06.839395 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:06.840144 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:06.840345 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:06.840397 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:06.843066 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57471: BlockingRecv error: recv got EOF from 127.0.98.1:57471 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":70,"threads_started":1}
Sep 26 09:28:06 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:06 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:06.975795 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:06.970796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.2:42067)
0926 09:28:06.970989 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:06.971002 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:06.971079 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:06.971193 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:06.971194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:06.971347 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:06.971392 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:06.971394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:06.972039 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:06.972040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:06.972639 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:06.972641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:06.973202 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:06.973203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:06.973265 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:06.973827 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:06.973832 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:06.975699 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":68,"threads_started":1}
W20260926 09:28:06.975837 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:06.970936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42067 (local address 127.0.98.62:40641)
0926 09:28:06.971012 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:06.971014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:06.971075 (+ 61us) server_negotiation.cc:407] Connection header received
0926 09:28:06.971124 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:06.971125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:06.971136 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:06.971182 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:06.971449 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:06.971955 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:06.972752 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:06.973125 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:06.973217 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:06.975776 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42067: BlockingRecv error: recv got EOF from 127.0.98.2:42067 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 09:28:07 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:07 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:07.097514 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:07.093079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.3:55805)
0926 09:28:07.093225 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:07.093235 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:07.093290 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:07.093448 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:07.093449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:07.093677 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:07.093728 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:07.093729 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:07.094376 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:07.094377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:07.094910 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:07.094911 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:07.095233 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:07.095234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:07.095276 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:07.095738 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:07.095742 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:07.097455 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
W20260926 09:28:07.097677 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:07.093138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55805 (local address 127.0.98.62:40641)
0926 09:28:07.093179 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:07.093181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:07.093289 (+ 108us) server_negotiation.cc:407] Connection header received
0926 09:28:07.093332 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:07.093333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:07.093346 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:07.093397 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:07.093812 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:07.094290 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:07.094983 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:07.095165 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:07.095218 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:07.097555 (+ 2337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55805: BlockingRecv error: recv got EOF from 127.0.98.3:55805 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 26 09:28:07 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:07 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:07.848323 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:07.843537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.1:46551)
0926 09:28:07.843672 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:07.843686 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:07.843744 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:07.843923 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:07.843924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:07.844079 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:07.844141 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:07.844142 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:07.844787 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:07.844787 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:07.845435 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:07.845438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:07.845950 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:07.845951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:07.845996 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:07.846474 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:07.846478 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:07.848265 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":104,"threads_started":1}
W20260926 09:28:07.848479 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:07.843658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46551 (local address 127.0.98.62:40641)
0926 09:28:07.843795 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:07.843797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:07.843805 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:28:07.843830 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:07.843830 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:07.843840 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:07.843906 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:07.844216 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:07.844720 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:07.845544 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:07.845882 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:07.845936 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:07.848387 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46551: BlockingRecv error: recv got EOF from 127.0.98.1:46551 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
Sep 26 09:28:07 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:07 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:07.981001 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:07.976326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.2:52077)
0926 09:28:07.976523 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:07.976534 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:07.976632 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:07.976805 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:07.976806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:07.977039 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:07.977098 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:07.977101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:07.977777 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:07.977778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:07.978307 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:07.978308 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:07.978643 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:07.978644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:07.978697 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:07.979222 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:07.979227 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:07.980941 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":91,"threads_started":1}
W20260926 09:28:07.981169 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:07.976428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52077 (local address 127.0.98.62:40641)
0926 09:28:07.976481 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:07.976488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:07.976629 (+ 141us) server_negotiation.cc:407] Connection header received
0926 09:28:07.976711 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:07.976712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:07.976728 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:07.976783 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:07.977201 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:07.977713 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:07.978380 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:07.978572 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:07.978628 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:07.981052 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52077: BlockingRecv error: recv got EOF from 127.0.98.2:52077 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 09:28:08 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:08 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:08.103108 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:08.098203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.3:59561)
0926 09:28:08.098372 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:08.098385 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:08.098443 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:08.098617 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:08.098618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:08.098886 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:08.098952 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:08.098954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:08.099668 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:08.099669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:08.100274 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:08.100275 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:08.100667 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:08.100668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:08.100734 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:08.101180 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:08.101185 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:08.102998 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":95,"threads_started":1}
W20260926 09:28:08.103175 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:08.098328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59561 (local address 127.0.98.62:40641)
0926 09:28:08.098379 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:08.098381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:08.098425 (+ 44us) server_negotiation.cc:407] Connection header received
0926 09:28:08.098539 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:08.098539 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:08.098551 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:08.098597 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:08.099034 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:08.099578 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:08.100364 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:08.100556 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:08.100646 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:08.103081 (+ 2435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59561: BlockingRecv error: recv got EOF from 127.0.98.3:59561 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 26 09:28:08 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:08 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:08.853444 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:08.848796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.1:43707)
0926 09:28:08.848955 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:08.848966 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:08.849020 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:08.849152 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:08.849152 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:08.849308 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:08.849355 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:08.849357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:08.850044 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:08.850045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:08.850594 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:08.850596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:08.850947 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:08.850948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:08.850989 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:08.851515 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:08.851520 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:08.853373 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
W20260926 09:28:08.853559 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:08.848890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43707 (local address 127.0.98.62:40641)
0926 09:28:08.848995 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:08.848997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:08.849006 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:28:08.849067 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:08.849067 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:08.849077 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:08.849129 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:08.849418 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:08.849954 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:08.850681 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:08.850882 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:08.850934 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:08.853478 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43707: BlockingRecv error: recv got EOF from 127.0.98.1:43707 (error 108)
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":49,"threads_started":1}
Sep 26 09:28:08 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:08 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:08.986632 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:08.981556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.2:57145)
0926 09:28:08.981729 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:08.981745 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:08.981825 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:08.982037 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:08.982038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:08.982254 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:08.982305 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:08.982307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:08.983020 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:08.983021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:08.983554 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:08.983556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:08.983952 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:08.983953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:08.984011 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:08.984561 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:08.984576 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:08.986506 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
W20260926 09:28:08.986675 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:08.981823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57145 (local address 127.0.98.62:40641)
0926 09:28:08.981898 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:08.981901 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:08.981910 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:28:08.981947 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:08.981947 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:08.981958 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:08.982031 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:08.982406 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:08.982923 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:08.983660 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:08.983863 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:08.983953 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:08.986606 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57145: BlockingRecv error: recv got EOF from 127.0.98.2:57145 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 09:28:09 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:09 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:09.108806 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:09.103783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.3:36005)
0926 09:28:09.103968 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:09.103980 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:09.104046 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:09.104217 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:09.104217 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:09.104378 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:09.104424 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:09.104425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:09.105099 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:09.105100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:09.105657 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:09.105659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:09.106045 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:09.106046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:09.106096 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:09.106547 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:09.106555 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:09.108710 (+ 2155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":102,"threads_started":1}
W20260926 09:28:09.108914 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:09.103852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36005 (local address 127.0.98.62:40641)
0926 09:28:09.103932 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:09.103936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:09.104050 (+ 114us) server_negotiation.cc:407] Connection header received
0926 09:28:09.104115 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:09.104116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:09.104131 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:09.104192 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:09.104505 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:09.105014 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:09.105754 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:09.105968 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:09.106031 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:09.108835 (+ 2804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36005: BlockingRecv error: recv got EOF from 127.0.98.3:36005 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 09:28:09 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:09 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:09.858594 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:09.853998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.1:35209)
0926 09:28:09.854167 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:09.854178 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:09.854230 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:09.854359 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:09.854360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:09.854534 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:09.854582 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:09.854584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:09.855218 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:09.855218 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:09.855744 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:09.855745 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:09.856067 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:09.856067 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:09.856106 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:09.856740 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:09.856745 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:09.858534 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":102,"threads_started":1}
W20260926 09:28:09.858733 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:09.854113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35209 (local address 127.0.98.62:40641)
0926 09:28:09.854241 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:09.854243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:09.854251 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:28:09.854273 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:09.854273 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:09.854283 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:09.854342 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:09.854653 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:09.855152 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:09.855812 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:09.856002 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:09.856054 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:09.858641 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35209: BlockingRecv error: recv got EOF from 127.0.98.1:35209 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
Sep 26 09:28:09 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:09 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:09.995242 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:09.987193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.2:48343)
0926 09:28:09.987417 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:09.987430 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:09.987533 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:09.987687 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:09.987687 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:09.987879 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:09.987925 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:09.987927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:09.988586 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:09.988588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:09.989112 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:09.989113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:09.989544 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:09.989545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:09.989640 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:09.993322 (+ 3682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:09.993335 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:09.995183 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":120,"threads_started":1}
W20260926 09:28:09.995399 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:09.987329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48343 (local address 127.0.98.62:40641)
0926 09:28:09.987402 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:09.987409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:09.987511 (+ 102us) server_negotiation.cc:407] Connection header received
0926 09:28:09.987596 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:09.987597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:09.987616 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:09.987681 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:09.988007 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:09.988500 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:09.989225 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:09.989441 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:09.989547 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:09.995323 (+ 5776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48343: BlockingRecv error: recv got EOF from 127.0.98.2:48343 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 09:28:10 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:10 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:10.114259 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:10.109682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.3:47863)
0926 09:28:10.109831 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:10.109852 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:10.109921 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:10.110056 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:10.110057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:10.110213 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:10.110263 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:10.110265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:10.111073 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:10.111074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:10.111615 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:10.111616 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:10.111935 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:10.111936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:10.111984 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:10.112425 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:10.112430 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:10.114188 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":66,"threads_started":1}
W20260926 09:28:10.114329 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:10.109775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47863 (local address 127.0.98.62:40641)
0926 09:28:10.109855 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:10.109857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:10.109904 (+ 47us) server_negotiation.cc:407] Connection header received
0926 09:28:10.109974 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:10.109975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:10.109987 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:10.110040 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:10.110339 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:10.110977 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:10.111686 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:10.111865 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:10.111917 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:10.114281 (+ 2364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47863: BlockingRecv error: recv got EOF from 127.0.98.3:47863 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 09:28:10 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:10 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:10.864006 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:10.859243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.1:59595)
0926 09:28:10.859400 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:10.859437 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:10.859491 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:10.859661 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:10.859661 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:10.859831 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:10.859882 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:10.859884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:10.860533 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:10.860534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:10.861124 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:10.861126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:10.861493 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:10.861494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:10.861543 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:10.862055 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:10.862060 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:10.863936 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
W20260926 09:28:10.864184 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:10.859346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59595 (local address 127.0.98.62:40641)
0926 09:28:10.859467 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:10.859469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:10.859486 (+ 17us) server_negotiation.cc:407] Connection header received
0926 09:28:10.859557 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:10.859557 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:10.859568 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:10.859638 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:10.859952 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:10.860447 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:10.861208 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:10.861392 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:10.861445 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:10.864096 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59595: BlockingRecv error: recv got EOF from 127.0.98.1:59595 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
Sep 26 09:28:10 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:28:10 dist-test-slave-dfzm krb5kdc[8516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:28:11.000727 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:10.995827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40641 (local address 127.0.98.2:50229)
0926 09:28:10.995941 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:10.995959 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:10.996016 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:10.996201 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:10.996202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:10.996383 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:10.996439 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:10.996442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:10.997110 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:10.997111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:10.997881 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:10.997883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:10.998262 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:10.998263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:10.998302 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:10.998794 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:10.998799 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:11.000659 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40641: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":67,"threads_started":1}
W20260926 09:28:11.000823 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:10.995924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50229 (local address 127.0.98.62:40641)
0926 09:28:10.995992 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:10.995998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:10.996010 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:28:10.996071 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:10.996072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:10.996082 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:10.996145 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:10.996533 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:10.997025 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:10.997972 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:10.998197 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:10.998254 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:11.000756 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50229: BlockingRecv error: recv got EOF from 127.0.98.2:50229 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 09:28:11.030156 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 9100
I20260926 09:28:11.034468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 9251
I20260926 09:28:11.038785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 9384
I20260926 09:28:11.043057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 9029
2026-09-26T09:28:11Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17451 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T09:28:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:28:11Z Disabled control of system clock
I20260926 09:28:11.070719 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37597
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 09:28:11.150991 9580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:28:11.151204 9580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:28:11.151225 9580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:28:11.152434 9580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:28:11.152477 9580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:28:11.152490 9580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:28:11.152503 9580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:28:11.152515 9580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:28:11.153846 9580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37597
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:28:11.154030 9580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:28:11.154273 9580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:28:11.159561 9588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:28:11.159565 9585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:28:11.159587 9586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:28:11.159601 9580 server_base.cc:1034] running on GCE node
I20260926 09:28:11.160032 9580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:28:11.160362 9580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:28:11.161530 9580 hybrid_clock.cc:648] HybridClock initialized: now 1790414891161507 us; error 36 us; skew 500 ppm
I20260926 09:28:11.162806 9580 webserver.cc:460] Webserver started at http://127.0.98.62:38755/ using document root <none> and password file <none>
I20260926 09:28:11.163075 9580 fs_manager.cc:359] Metadata directory not provided
I20260926 09:28:11.163147 9580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:28:11.163295 9580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:28:11.164337 9580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "d0d754d640dd4907b2c7cf00c82bc330"
format_stamp: "Formatted at 2026-09-26 09:28:11 on dist-test-slave-dfzm"
server_key: "674b2e61316de55674a85695ab72598b"
server_key_iv: "3d88461ba03b71d23aa91f7b0e698379"
server_key_version: "encryptionkey@0"
I20260926 09:28:11.164706 9580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "d0d754d640dd4907b2c7cf00c82bc330"
format_stamp: "Formatted at 2026-09-26 09:28:11 on dist-test-slave-dfzm"
server_key: "674b2e61316de55674a85695ab72598b"
server_key_iv: "3d88461ba03b71d23aa91f7b0e698379"
server_key_version: "encryptionkey@0"
I20260926 09:28:11.166054 9580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:28:11.166767 9594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:11.166937 9580 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:28:11.167032 9580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "d0d754d640dd4907b2c7cf00c82bc330"
format_stamp: "Formatted at 2026-09-26 09:28:11 on dist-test-slave-dfzm"
server_key: "674b2e61316de55674a85695ab72598b"
server_key_iv: "3d88461ba03b71d23aa91f7b0e698379"
server_key_version: "encryptionkey@0"
I20260926 09:28:11.167117 9580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:28:11.176368 9580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:28:11.176677 9580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:28:11.176827 9580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:28:11.180788 9580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37597
I20260926 09:28:11.180871 9646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37597 every 8 connection(s)
I20260926 09:28:11.181222 9580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:28:11.181947 9647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:28:11.184650 9647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0d754d640dd4907b2c7cf00c82bc330: Bootstrap starting.
I20260926 09:28:11.185014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 9580
I20260926 09:28:11.185128 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:28:11.185289 392 external_mini_cluster.cc:1428] Setting key 4d61044b1b47cf7c5e827cbf815873a1
I20260926 09:28:11.185374 9647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0d754d640dd4907b2c7cf00c82bc330: Neither blocks nor log segments found. Creating new log.
I20260926 09:28:11.185746 9647 log.cc:824] T 00000000000000000000000000000000 P d0d754d640dd4907b2c7cf00c82bc330: Log is configured to *not* fsync() on all Append() calls
I20260926 09:28:11.186554 9647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0d754d640dd4907b2c7cf00c82bc330: No bootstrap required, opened a new log
I20260926 09:28:11.188719 9647 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0d754d640dd4907b2c7cf00c82bc330 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d754d640dd4907b2c7cf00c82bc330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37597 } }
I20260926 09:28:11.188922 9647 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0d754d640dd4907b2c7cf00c82bc330 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:28:11.188961 9647 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0d754d640dd4907b2c7cf00c82bc330 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d754d640dd4907b2c7cf00c82bc330, State: Initialized, Role: FOLLOWER
I20260926 09:28:11.189100 9647 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0d754d640dd4907b2c7cf00c82bc330 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d754d640dd4907b2c7cf00c82bc330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37597 } }
I20260926 09:28:11.189200 9647 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0d754d640dd4907b2c7cf00c82bc330 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:28:11.189231 9647 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0d754d640dd4907b2c7cf00c82bc330 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:28:11.189267 9647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0d754d640dd4907b2c7cf00c82bc330 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:28:11.190035 9647 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0d754d640dd4907b2c7cf00c82bc330 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d754d640dd4907b2c7cf00c82bc330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37597 } }
I20260926 09:28:11.190179 9647 leader_election.cc:302] T 00000000000000000000000000000000 P d0d754d640dd4907b2c7cf00c82bc330 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0d754d640dd4907b2c7cf00c82bc330; no voters:
I20260926 09:28:11.190351 9647 leader_election.cc:288] T 00000000000000000000000000000000 P d0d754d640dd4907b2c7cf00c82bc330 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:28:11.190429 9652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0d754d640dd4907b2c7cf00c82bc330 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:28:11.190621 9652 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0d754d640dd4907b2c7cf00c82bc330 [term 1 LEADER]: Becoming Leader. State: Replica: d0d754d640dd4907b2c7cf00c82bc330, State: Running, Role: LEADER
I20260926 09:28:11.190645 9647 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0d754d640dd4907b2c7cf00c82bc330 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:28:11.190781 9652 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0d754d640dd4907b2c7cf00c82bc330 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d754d640dd4907b2c7cf00c82bc330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37597 } }
I20260926 09:28:11.191255 9653 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0d754d640dd4907b2c7cf00c82bc330 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0d754d640dd4907b2c7cf00c82bc330" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d754d640dd4907b2c7cf00c82bc330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37597 } } }
I20260926 09:28:11.191345 9653 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0d754d640dd4907b2c7cf00c82bc330 [sys.catalog]: This master's current role is: LEADER
I20260926 09:28:11.191515 9654 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0d754d640dd4907b2c7cf00c82bc330 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0d754d640dd4907b2c7cf00c82bc330. Latest consensus state: current_term: 1 leader_uuid: "d0d754d640dd4907b2c7cf00c82bc330" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d754d640dd4907b2c7cf00c82bc330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37597 } } }
I20260926 09:28:11.191535 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:11.185838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56994 (local address 127.0.98.62:37597)
0926 09:28:11.186034 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:11.186044 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:11.186103 (+ 59us) server_negotiation.cc:407] Connection header received
0926 09:28:11.186583 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:11.186590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:11.186648 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:11.186806 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:11.187039 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:11.187893 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:11.189079 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:11.189729 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:11.189865 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:11.190687 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:28:11.190692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:28:11.190692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:28:11.190936 (+ 244us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:28:11.190978 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:28:11.190993 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:28:11.191036 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:11.191193 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 09:28:11.191291 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260926 09:28:11.191578 9661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:28:11.191635 9654 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0d754d640dd4907b2c7cf00c82bc330 [sys.catalog]: This master's current role is: LEADER
I20260926 09:28:11.192550 9661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:28:11.194037 9661 catalog_manager.cc:1347] Generated new cluster ID: c9ee498c9bcb4fa1ac506ef81e9a72be
I20260926 09:28:11.194118 9661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:28:11.204207 9661 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:28:11.204938 9661 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:28:11.214133 9661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0d754d640dd4907b2c7cf00c82bc330: Generated new TSK 0
I20260926 09:28:11.214359 9661 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:28:11.215996 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37597
--builtin_ntp_servers=127.0.98.20:33313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 09:28:11.297565 9671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:28:11.297894 9671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:28:11.297932 9671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:28:11.299155 9671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:28:11.299223 9671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:28:11.299268 9671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:28:11.300757 9671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:28:11.301038 9671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:28:11.301360 9671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:28:11.306548 9677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:28:11.306569 9676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:28:11.306748 9671 server_base.cc:1034] running on GCE node
W20260926 09:28:11.306849 9679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:28:11.307041 9671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:28:11.307325 9671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:28:11.308493 9671 hybrid_clock.cc:648] HybridClock initialized: now 1790414891308477 us; error 31 us; skew 500 ppm
I20260926 09:28:11.309829 9671 webserver.cc:460] Webserver started at http://127.0.98.1:41989/ using document root <none> and password file <none>
I20260926 09:28:11.310067 9671 fs_manager.cc:359] Metadata directory not provided
I20260926 09:28:11.310145 9671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:28:11.310298 9671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:28:11.311447 9671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "cf41302723204660911bf92759b5aa4d"
format_stamp: "Formatted at 2026-09-26 09:28:11 on dist-test-slave-dfzm"
server_key: "e4da5b2ce0f91651351a29554b4b20dd"
server_key_iv: "6e0fbb912cb4fcc1a20f60198e6f4d9f"
server_key_version: "encryptionkey@0"
I20260926 09:28:11.311803 9671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cf41302723204660911bf92759b5aa4d"
format_stamp: "Formatted at 2026-09-26 09:28:11 on dist-test-slave-dfzm"
server_key: "e4da5b2ce0f91651351a29554b4b20dd"
server_key_iv: "6e0fbb912cb4fcc1a20f60198e6f4d9f"
server_key_version: "encryptionkey@0"
I20260926 09:28:11.313198 9671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:28:11.314033 9685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:11.314198 9671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:11.314302 9671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "cf41302723204660911bf92759b5aa4d"
format_stamp: "Formatted at 2026-09-26 09:28:11 on dist-test-slave-dfzm"
server_key: "e4da5b2ce0f91651351a29554b4b20dd"
server_key_iv: "6e0fbb912cb4fcc1a20f60198e6f4d9f"
server_key_version: "encryptionkey@0"
I20260926 09:28:11.314376 9671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:28:11.347064 9671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:28:11.347414 9671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:28:11.347576 9671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:28:11.347839 9671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:28:11.348199 9671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:28:11.348254 9671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:11.348297 9671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:28:11.348326 9671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:11.354537 9671 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37685
I20260926 09:28:11.354578 9798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37685 every 8 connection(s)
I20260926 09:28:11.354923 9671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:28:11.360162 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:11.355707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41469 (local address 127.0.98.62:37597)
0926 09:28:11.355771 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:11.355773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:11.356005 (+ 232us) server_negotiation.cc:407] Connection header received
0926 09:28:11.356569 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:11.356570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:11.356582 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:11.356648 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:11.357290 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:11.357890 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:11.358905 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:11.359094 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:11.359145 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:11.359919 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:28:11.359920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:28:11.359921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:28:11.359925 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:28:11.359940 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:28:11.359943 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:28:11.359987 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:11.360073 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 09:28:11.360099 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:28:11.360378 9801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:11.355688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37597 (local address 127.0.98.1:41469)
0926 09:28:11.355873 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:11.355914 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:11.356442 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:11.356703 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:11.356706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:11.356714 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:11.357192 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:11.357196 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:11.358000 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:11.358002 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:11.358822 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:11.358824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:11.359166 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:11.359166 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:11.359275 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:11.359787 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:28:11.359788 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:11.359817 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:28:11.359839 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:28:11.359997 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:28:11.360001 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:28:11.360004 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:28:11.360056 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 09:28:11.360134 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260926 09:28:11.360931 9799 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37597
I20260926 09:28:11.361024 9799 heartbeater.cc:464] Registering TS with master...
I20260926 09:28:11.361150 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 9671
I20260926 09:28:11.361173 9799 heartbeater.cc:511] Master 127.0.98.62:37597 requested a full tablet report, sending...
I20260926 09:28:11.361213 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:28:11.361361 392 external_mini_cluster.cc:1428] Setting key cef07106cad33c7b1f30037f61610af7
I20260926 09:28:11.361833 9611 ts_manager.cc:194] Registered new tserver with Master: cf41302723204660911bf92759b5aa4d (127.0.98.1:37685)
I20260926 09:28:11.362565 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37597
--builtin_ntp_servers=127.0.98.20:33313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 09:28:11.362577 9611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:41469
W20260926 09:28:11.445295 9802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:28:11.445551 9802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:28:11.445587 9802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:28:11.446763 9802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:28:11.446837 9802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:28:11.446883 9802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:28:11.448320 9802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:28:11.448603 9802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:28:11.448913 9802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:28:11.454110 9802 server_base.cc:1034] running on GCE node
W20260926 09:28:11.454093 9807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:28:11.454082 9810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:28:11.454087 9808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:28:11.454495 9802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:28:11.454741 9802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:28:11.455897 9802 hybrid_clock.cc:648] HybridClock initialized: now 1790414891455868 us; error 47 us; skew 500 ppm
I20260926 09:28:11.457115 9802 webserver.cc:460] Webserver started at http://127.0.98.2:34287/ using document root <none> and password file <none>
I20260926 09:28:11.457360 9802 fs_manager.cc:359] Metadata directory not provided
I20260926 09:28:11.457428 9802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:28:11.457563 9802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:28:11.458665 9802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "9beda55ecbbe45b5be0e85c9931c1783"
format_stamp: "Formatted at 2026-09-26 09:28:11 on dist-test-slave-dfzm"
server_key: "1625dbfb8b5f6f8ae4f0b412df61289d"
server_key_iv: "b57aa8583ea23ab0cfa0b76e4fb99366"
server_key_version: "encryptionkey@0"
I20260926 09:28:11.459025 9802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9beda55ecbbe45b5be0e85c9931c1783"
format_stamp: "Formatted at 2026-09-26 09:28:11 on dist-test-slave-dfzm"
server_key: "1625dbfb8b5f6f8ae4f0b412df61289d"
server_key_iv: "b57aa8583ea23ab0cfa0b76e4fb99366"
server_key_version: "encryptionkey@0"
I20260926 09:28:11.460309 9802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:28:11.461026 9816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:11.461189 9802 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:28:11.461284 9802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "9beda55ecbbe45b5be0e85c9931c1783"
format_stamp: "Formatted at 2026-09-26 09:28:11 on dist-test-slave-dfzm"
server_key: "1625dbfb8b5f6f8ae4f0b412df61289d"
server_key_iv: "b57aa8583ea23ab0cfa0b76e4fb99366"
server_key_version: "encryptionkey@0"
I20260926 09:28:11.461369 9802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:28:11.481190 9802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:28:11.481462 9802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:28:11.481561 9802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:28:11.481828 9802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:28:11.482187 9802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:28:11.482221 9802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:11.482277 9802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:28:11.482303 9802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:11.488250 9802 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40669
I20260926 09:28:11.488329 9929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40669 every 8 connection(s)
I20260926 09:28:11.488667 9802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:28:11.493928 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:11.489403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59991 (local address 127.0.98.62:37597)
0926 09:28:11.489463 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:11.489465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:11.489722 (+ 257us) server_negotiation.cc:407] Connection header received
0926 09:28:11.490251 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:11.490252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:11.490261 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:11.490324 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:11.491006 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:11.491570 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:11.492700 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:11.492896 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:11.492953 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:11.493739 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:28:11.493740 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:28:11.493740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:28:11.493743 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:28:11.493756 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:28:11.493760 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:28:11.493803 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:11.493850 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 09:28:11.493870 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 09:28:11.494179 9932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:11.489388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37597 (local address 127.0.98.2:59991)
0926 09:28:11.489563 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:11.489624 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:11.490141 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:11.490378 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:11.490382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:11.490390 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:11.490922 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:11.490933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:11.491655 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:11.491656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:11.492519 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:11.492522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:11.492967 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:11.492967 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:11.493069 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:11.493556 (+ 487us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:28:11.493557 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:11.493628 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:28:11.493663 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:28:11.493803 (+ 140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:28:11.493805 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:28:11.493808 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:28:11.493852 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 09:28:11.493926 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260926 09:28:11.494750 9930 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37597
I20260926 09:28:11.494861 9930 heartbeater.cc:464] Registering TS with master...
I20260926 09:28:11.495044 9930 heartbeater.cc:511] Master 127.0.98.62:37597 requested a full tablet report, sending...
I20260926 09:28:11.495402 9611 ts_manager.cc:194] Registered new tserver with Master: 9beda55ecbbe45b5be0e85c9931c1783 (127.0.98.2:40669)
I20260926 09:28:11.495790 9611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:59991
I20260926 09:28:11.497924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 9802
I20260926 09:28:11.498006 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:28:11.498220 392 external_mini_cluster.cc:1428] Setting key 3c0ff1d1a17545a0ceda9e38f54b02b7
I20260926 09:28:11.499118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37597
--builtin_ntp_servers=127.0.98.20:33313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 09:28:11.579572 9933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:28:11.579780 9933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:28:11.579802 9933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:28:11.580962 9933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:28:11.581005 9933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:28:11.581030 9933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:28:11.582391 9933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:28:11.582646 9933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:28:11.582944 9933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:28:11.587950 9933 server_base.cc:1034] running on GCE node
W20260926 09:28:11.587935 9938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:28:11.588092 9939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:28:11.587925 9941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:28:11.588440 9933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:28:11.588699 9933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:28:11.589874 9933 hybrid_clock.cc:648] HybridClock initialized: now 1790414891589837 us; error 54 us; skew 500 ppm
I20260926 09:28:11.591317 9933 webserver.cc:460] Webserver started at http://127.0.98.3:46639/ using document root <none> and password file <none>
I20260926 09:28:11.591583 9933 fs_manager.cc:359] Metadata directory not provided
I20260926 09:28:11.591660 9933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:28:11.591830 9933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:28:11.592928 9933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "5cf1e29377f548c88558e6f1ffd9b536"
format_stamp: "Formatted at 2026-09-26 09:28:11 on dist-test-slave-dfzm"
server_key: "f01b768d460ea0efcf43ec669388ca41"
server_key_iv: "cf115028ab5e7fd03170367525055fa9"
server_key_version: "encryptionkey@0"
I20260926 09:28:11.593292 9933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5cf1e29377f548c88558e6f1ffd9b536"
format_stamp: "Formatted at 2026-09-26 09:28:11 on dist-test-slave-dfzm"
server_key: "f01b768d460ea0efcf43ec669388ca41"
server_key_iv: "cf115028ab5e7fd03170367525055fa9"
server_key_version: "encryptionkey@0"
I20260926 09:28:11.594686 9933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:28:11.595458 9947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:11.595666 9933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:28:11.595765 9933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "5cf1e29377f548c88558e6f1ffd9b536"
format_stamp: "Formatted at 2026-09-26 09:28:11 on dist-test-slave-dfzm"
server_key: "f01b768d460ea0efcf43ec669388ca41"
server_key_iv: "cf115028ab5e7fd03170367525055fa9"
server_key_version: "encryptionkey@0"
I20260926 09:28:11.595856 9933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:28:11.611575 9933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:28:11.611923 9933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:28:11.612079 9933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:28:11.612315 9933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:28:11.612658 9933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:28:11.612712 9933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:11.612756 9933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:28:11.612782 9933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:11.618795 9933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33273
I20260926 09:28:11.618891 10060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33273 every 8 connection(s)
I20260926 09:28:11.619201 9933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:28:11.623796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 9933
I20260926 09:28:11.623939 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:28:11.624142 392 external_mini_cluster.cc:1428] Setting key da315ca76c248ac5e569c64cb9a2e06b
I20260926 09:28:11.624742 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:11.620021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47397 (local address 127.0.98.62:37597)
0926 09:28:11.620083 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:11.620085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:11.620343 (+ 258us) server_negotiation.cc:407] Connection header received
0926 09:28:11.620809 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:11.620810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:11.620819 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:11.620874 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:11.621529 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:11.622076 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:11.623069 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:11.623250 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:11.623300 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:11.624327 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:28:11.624329 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:28:11.624329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:28:11.624336 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:28:11.624355 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:28:11.624360 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:28:11.624421 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:11.624642 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 09:28:11.624666 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 09:28:11.624929 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:11.619996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37597 (local address 127.0.98.3:47397)
0926 09:28:11.620207 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:11.620249 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:11.620729 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:11.620933 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:11.620937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:11.620955 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:11.621460 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:11.621464 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:11.622169 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:11.622170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:11.622989 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:11.622992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:11.623323 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:11.623323 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:11.623430 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:11.624109 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:28:11.624110 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:11.624149 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:28:11.624178 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:28:11.624494 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:28:11.624498 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:28:11.624501 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:28:11.624591 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 09:28:11.624699 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":110,"threads_started":1}
I20260926 09:28:11.625383 10061 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37597
I20260926 09:28:11.625463 10061 heartbeater.cc:464] Registering TS with master...
I20260926 09:28:11.625651 10061 heartbeater.cc:511] Master 127.0.98.62:37597 requested a full tablet report, sending...
I20260926 09:28:11.625994 9611 ts_manager.cc:194] Registered new tserver with Master: 5cf1e29377f548c88558e6f1ffd9b536 (127.0.98.3:33273)
I20260926 09:28:11.626324 9611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:47397
I20260926 09:28:11.636080 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:28:11.638271 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 9671
W20260926 09:28:11.638293 9650 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:11.637642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57002 (local address 127.0.98.62:37597)
0926 09:28:11.637702 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:11.637704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:11.637786 (+ 82us) server_negotiation.cc:407] Connection header received
0926 09:28:11.637824 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:11.637825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:11.637834 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:11.637887 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:11.638023 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57002: BlockingRecv error: recv got EOF from 127.0.0.1:57002 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 09:28:11.642601 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 9802
I20260926 09:28:11.646638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 9933
I20260926 09:28:11.650839 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 9580
2026-09-26T09:28:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (606 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T09:28:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:28:11Z Disabled control of system clock
I20260926 09:28:11.677408 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45243
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 09:28:11.759022 10072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:28:11.759241 10072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:28:11.759263 10072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:28:11.760376 10072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:28:11.760413 10072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:28:11.760426 10072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:28:11.760439 10072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:28:11.760452 10072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:28:11.761919 10072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45243
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:28:11.762135 10072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:28:11.762378 10072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:28:11.767287 10080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:28:11.767323 10072 server_base.cc:1034] running on GCE node
W20260926 09:28:11.767287 10077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:28:11.767302 10078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:28:11.767845 10072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:28:11.768193 10072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:28:11.769357 10072 hybrid_clock.cc:648] HybridClock initialized: now 1790414891769348 us; error 41 us; skew 500 ppm
I20260926 09:28:11.770655 10072 webserver.cc:460] Webserver started at http://127.0.98.62:37193/ using document root <none> and password file <none>
I20260926 09:28:11.770917 10072 fs_manager.cc:359] Metadata directory not provided
I20260926 09:28:11.770992 10072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:28:11.771152 10072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:28:11.772317 10072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "534d2f1a31394d9da0a9254683cdcdbe"
format_stamp: "Formatted at 2026-09-26 09:28:11 on dist-test-slave-dfzm"
server_key: "1a107e669cb1b4b66c7b8f014bfd77ef"
server_key_iv: "bae978b91e047b074c7db3f2a1755e8b"
server_key_version: "encryptionkey@0"
I20260926 09:28:11.772696 10072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "534d2f1a31394d9da0a9254683cdcdbe"
format_stamp: "Formatted at 2026-09-26 09:28:11 on dist-test-slave-dfzm"
server_key: "1a107e669cb1b4b66c7b8f014bfd77ef"
server_key_iv: "bae978b91e047b074c7db3f2a1755e8b"
server_key_version: "encryptionkey@0"
I20260926 09:28:11.774101 10072 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:28:11.774796 10086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:11.774945 10072 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:28:11.775051 10072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "534d2f1a31394d9da0a9254683cdcdbe"
format_stamp: "Formatted at 2026-09-26 09:28:11 on dist-test-slave-dfzm"
server_key: "1a107e669cb1b4b66c7b8f014bfd77ef"
server_key_iv: "bae978b91e047b074c7db3f2a1755e8b"
server_key_version: "encryptionkey@0"
I20260926 09:28:11.775126 10072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:28:11.778525 10072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:28:11.778802 10072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:28:11.778950 10072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:28:11.782846 10072 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45243
I20260926 09:28:11.782881 10138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45243 every 8 connection(s)
I20260926 09:28:11.783206 10072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:28:11.783825 10139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:28:11.786684 10139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 534d2f1a31394d9da0a9254683cdcdbe: Bootstrap starting.
I20260926 09:28:11.787277 10139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 534d2f1a31394d9da0a9254683cdcdbe: Neither blocks nor log segments found. Creating new log.
I20260926 09:28:11.787580 10139 log.cc:824] T 00000000000000000000000000000000 P 534d2f1a31394d9da0a9254683cdcdbe: Log is configured to *not* fsync() on all Append() calls
I20260926 09:28:11.788335 10139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 534d2f1a31394d9da0a9254683cdcdbe: No bootstrap required, opened a new log
I20260926 09:28:11.790092 10139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 534d2f1a31394d9da0a9254683cdcdbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534d2f1a31394d9da0a9254683cdcdbe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45243 } }
I20260926 09:28:11.790238 10139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 534d2f1a31394d9da0a9254683cdcdbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:28:11.790261 10139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 534d2f1a31394d9da0a9254683cdcdbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 534d2f1a31394d9da0a9254683cdcdbe, State: Initialized, Role: FOLLOWER
I20260926 09:28:11.790432 10139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 534d2f1a31394d9da0a9254683cdcdbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534d2f1a31394d9da0a9254683cdcdbe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45243 } }
I20260926 09:28:11.790509 10139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 534d2f1a31394d9da0a9254683cdcdbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:28:11.790553 10139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 534d2f1a31394d9da0a9254683cdcdbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:28:11.790601 10139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 534d2f1a31394d9da0a9254683cdcdbe [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:28:11.791163 10139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 534d2f1a31394d9da0a9254683cdcdbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534d2f1a31394d9da0a9254683cdcdbe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45243 } }
I20260926 09:28:11.791252 10139 leader_election.cc:302] T 00000000000000000000000000000000 P 534d2f1a31394d9da0a9254683cdcdbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 534d2f1a31394d9da0a9254683cdcdbe; no voters:
I20260926 09:28:11.791443 10139 leader_election.cc:288] T 00000000000000000000000000000000 P 534d2f1a31394d9da0a9254683cdcdbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:28:11.791498 10142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 534d2f1a31394d9da0a9254683cdcdbe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:28:11.791744 10139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 534d2f1a31394d9da0a9254683cdcdbe [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:28:11.791759 10142 raft_consensus.cc:686] T 00000000000000000000000000000000 P 534d2f1a31394d9da0a9254683cdcdbe [term 1 LEADER]: Becoming Leader. State: Replica: 534d2f1a31394d9da0a9254683cdcdbe, State: Running, Role: LEADER
I20260926 09:28:11.791939 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 10072
I20260926 09:28:11.791985 10142 consensus_queue.cc:237] T 00000000000000000000000000000000 P 534d2f1a31394d9da0a9254683cdcdbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534d2f1a31394d9da0a9254683cdcdbe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45243 } }
I20260926 09:28:11.792038 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:28:11.792235 392 external_mini_cluster.cc:1428] Setting key 303a544cb69b9e9c4651a52b61d75dc5
I20260926 09:28:11.792632 10143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 534d2f1a31394d9da0a9254683cdcdbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "534d2f1a31394d9da0a9254683cdcdbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534d2f1a31394d9da0a9254683cdcdbe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45243 } } }
I20260926 09:28:11.792788 10143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 534d2f1a31394d9da0a9254683cdcdbe [sys.catalog]: This master's current role is: LEADER
I20260926 09:28:11.793115 10144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 534d2f1a31394d9da0a9254683cdcdbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 534d2f1a31394d9da0a9254683cdcdbe. Latest consensus state: current_term: 1 leader_uuid: "534d2f1a31394d9da0a9254683cdcdbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534d2f1a31394d9da0a9254683cdcdbe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45243 } } }
I20260926 09:28:11.793179 10153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:28:11.793246 10144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 534d2f1a31394d9da0a9254683cdcdbe [sys.catalog]: This master's current role is: LEADER
I20260926 09:28:11.794333 10153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:28:11.794992 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:11.792883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52548 (local address 127.0.98.62:45243)
0926 09:28:11.793301 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:11.793310 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:11.793349 (+ 39us) server_negotiation.cc:407] Connection header received
0926 09:28:11.793488 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:11.793496 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:11.793547 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:11.793751 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:11.794520 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:28:11.794523 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:28:11.794523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:28:11.794527 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:28:11.794563 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:28:11.794574 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:28:11.794655 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:11.794726 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 09:28:11.794791 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":61,"threads_started":1}
I20260926 09:28:11.795887 10153 catalog_manager.cc:1347] Generated new cluster ID: 975e8731125b4ef9b0326c312806a976
I20260926 09:28:11.795938 10153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:28:11.828151 10153 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:28:11.828351 10153 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:28:11.836989 10153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 534d2f1a31394d9da0a9254683cdcdbe: Generated new TSK 0
I20260926 09:28:11.837203 10153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:28:11.839852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45243
--builtin_ntp_servers=127.0.98.20:35691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 09:28:11.920671 10163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:28:11.920876 10163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:28:11.920899 10163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:28:11.922117 10163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:28:11.922158 10163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:28:11.922184 10163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:28:11.923549 10163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:28:11.923815 10163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:28:11.924170 10163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:28:11.929190 10163 server_base.cc:1034] running on GCE node
W20260926 09:28:11.929142 10168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:28:11.929157 10169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:28:11.929127 10171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:28:11.929566 10163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:28:11.929839 10163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:28:11.931001 10163 hybrid_clock.cc:648] HybridClock initialized: now 1790414891930991 us; error 42 us; skew 500 ppm
I20260926 09:28:11.932271 10163 webserver.cc:460] Webserver started at http://127.0.98.1:39943/ using document root <none> and password file <none>
I20260926 09:28:11.932510 10163 fs_manager.cc:359] Metadata directory not provided
I20260926 09:28:11.932580 10163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:28:11.932746 10163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:28:11.933877 10163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "9151fd33ea144591a654746d716bb543"
format_stamp: "Formatted at 2026-09-26 09:28:11 on dist-test-slave-dfzm"
server_key: "f9cbfdc70efc3b486cf6076a7d6caceb"
server_key_iv: "3fa16a776a08a0a71e0c106614e53a69"
server_key_version: "encryptionkey@0"
I20260926 09:28:11.934255 10163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9151fd33ea144591a654746d716bb543"
format_stamp: "Formatted at 2026-09-26 09:28:11 on dist-test-slave-dfzm"
server_key: "f9cbfdc70efc3b486cf6076a7d6caceb"
server_key_iv: "3fa16a776a08a0a71e0c106614e53a69"
server_key_version: "encryptionkey@0"
I20260926 09:28:11.935531 10163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:28:11.936201 10177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:11.936350 10163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:28:11.936450 10163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "9151fd33ea144591a654746d716bb543"
format_stamp: "Formatted at 2026-09-26 09:28:11 on dist-test-slave-dfzm"
server_key: "f9cbfdc70efc3b486cf6076a7d6caceb"
server_key_iv: "3fa16a776a08a0a71e0c106614e53a69"
server_key_version: "encryptionkey@0"
I20260926 09:28:11.936528 10163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:28:11.939891 10163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:28:11.940143 10163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:28:11.940286 10163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:28:11.940491 10163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:28:11.940793 10163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:28:11.940847 10163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:11.940886 10163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:28:11.940913 10163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:11.946635 10163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40871
I20260926 09:28:11.946671 10290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40871 every 8 connection(s)
I20260926 09:28:11.947023 10163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:28:11.950093 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:11.947787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37979 (local address 127.0.98.62:45243)
0926 09:28:11.947853 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:11.947855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:11.948085 (+ 230us) server_negotiation.cc:407] Connection header received
0926 09:28:11.948598 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:11.948600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:11.948608 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:11.948658 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:11.949817 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:28:11.949818 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:28:11.949819 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:28:11.949823 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:28:11.949840 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:28:11.949845 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:28:11.949906 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:11.950003 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 09:28:11.950021 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:28:11.950321 10293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:11.947792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45243 (local address 127.0.98.1:37979)
0926 09:28:11.947959 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:11.947994 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:11.948486 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:11.948735 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:11.948738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:11.948746 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:11.949652 (+ 906us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:28:11.949652 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:11.949681 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:28:11.949710 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:28:11.949912 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:28:11.949917 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:28:11.949919 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:28:11.950014 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 09:28:11.950076 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20260926 09:28:11.950862 10291 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45243
I20260926 09:28:11.950956 10291 heartbeater.cc:464] Registering TS with master...
I20260926 09:28:11.951107 10291 heartbeater.cc:511] Master 127.0.98.62:45243 requested a full tablet report, sending...
I20260926 09:28:11.951625 10103 ts_manager.cc:194] Registered new tserver with Master: 9151fd33ea144591a654746d716bb543 (127.0.98.1:40871)
I20260926 09:28:11.954675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 10163
I20260926 09:28:11.954761 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:28:11.954924 392 external_mini_cluster.cc:1428] Setting key d3e1d7ed24d6116246dc2d40574686c1
I20260926 09:28:11.955912 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45243
--builtin_ntp_servers=127.0.98.20:35691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 09:28:12.039260 10294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:28:12.039465 10294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:28:12.039495 10294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:28:12.040704 10294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:28:12.040742 10294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:28:12.040768 10294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:28:12.042107 10294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:28:12.042294 10294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:28:12.042546 10294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:28:12.047528 10300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:28:12.047564 10299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:28:12.047658 10294 server_base.cc:1034] running on GCE node
W20260926 09:28:12.047537 10302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:28:12.047981 10294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:28:12.048190 10294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:28:12.049329 10294 hybrid_clock.cc:648] HybridClock initialized: now 1790414892049322 us; error 29 us; skew 500 ppm
I20260926 09:28:12.050719 10294 webserver.cc:460] Webserver started at http://127.0.98.2:36219/ using document root <none> and password file <none>
I20260926 09:28:12.050952 10294 fs_manager.cc:359] Metadata directory not provided
I20260926 09:28:12.050998 10294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:28:12.051165 10294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:28:12.052289 10294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "0b88fd91a4894aa99ee7cfb7a5afd599"
format_stamp: "Formatted at 2026-09-26 09:28:12 on dist-test-slave-dfzm"
server_key: "481aca6fd842c7e5086c0432f2dbd7ed"
server_key_iv: "7b84a41ee14c874f2bde577453cdc9d4"
server_key_version: "encryptionkey@0"
I20260926 09:28:12.052644 10294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0b88fd91a4894aa99ee7cfb7a5afd599"
format_stamp: "Formatted at 2026-09-26 09:28:12 on dist-test-slave-dfzm"
server_key: "481aca6fd842c7e5086c0432f2dbd7ed"
server_key_iv: "7b84a41ee14c874f2bde577453cdc9d4"
server_key_version: "encryptionkey@0"
I20260926 09:28:12.054006 10294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:28:12.054730 10308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:12.054904 10294 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:28:12.055042 10294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "0b88fd91a4894aa99ee7cfb7a5afd599"
format_stamp: "Formatted at 2026-09-26 09:28:12 on dist-test-slave-dfzm"
server_key: "481aca6fd842c7e5086c0432f2dbd7ed"
server_key_iv: "7b84a41ee14c874f2bde577453cdc9d4"
server_key_version: "encryptionkey@0"
I20260926 09:28:12.055120 10294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:28:12.058533 10294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:28:12.058794 10294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:28:12.058936 10294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:28:12.059151 10294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:28:12.059453 10294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:28:12.059505 10294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:12.059545 10294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:28:12.059572 10294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:12.065434 10294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42729
I20260926 09:28:12.065471 10421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42729 every 8 connection(s)
I20260926 09:28:12.065855 10294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:28:12.069051 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:12.066652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55613 (local address 127.0.98.62:45243)
0926 09:28:12.066755 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:12.066757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:12.066980 (+ 223us) server_negotiation.cc:407] Connection header received
0926 09:28:12.067486 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:12.067487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:12.067493 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:12.067543 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:12.068775 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:28:12.068777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:28:12.068777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:28:12.068782 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:28:12.068799 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:28:12.068804 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:28:12.068879 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:12.068959 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 09:28:12.068974 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 09:28:12.069252 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:12.066607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45243 (local address 127.0.98.2:55613)
0926 09:28:12.066836 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:12.066880 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:12.067375 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:12.067613 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:12.067617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:12.067627 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:12.068587 (+ 960us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:28:12.068588 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:12.068617 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:28:12.068646 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:28:12.068889 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:28:12.068896 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:28:12.068899 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:28:12.068972 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 09:28:12.069037 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":97,"threads_started":1}
I20260926 09:28:12.069830 10422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45243
I20260926 09:28:12.069911 10422 heartbeater.cc:464] Registering TS with master...
I20260926 09:28:12.070103 10422 heartbeater.cc:511] Master 127.0.98.62:45243 requested a full tablet report, sending...
I20260926 09:28:12.070515 10103 ts_manager.cc:194] Registered new tserver with Master: 0b88fd91a4894aa99ee7cfb7a5afd599 (127.0.98.2:42729)
I20260926 09:28:12.070756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 10294
I20260926 09:28:12.070881 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:28:12.071069 392 external_mini_cluster.cc:1428] Setting key 6230e045f268edcf22462e18d8f1fdc7
I20260926 09:28:12.071933 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45243
--builtin_ntp_servers=127.0.98.20:35691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 09:28:12.152909 10425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:28:12.153115 10425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:28:12.153137 10425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:28:12.154397 10425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:28:12.154441 10425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:28:12.154469 10425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:28:12.155839 10425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:28:12.156085 10425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:28:12.156384 10425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:28:12.161389 10430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:28:12.161393 10431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:28:12.161394 10433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:28:12.161700 10425 server_base.cc:1034] running on GCE node
I20260926 09:28:12.161917 10425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:28:12.162186 10425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:28:12.163342 10425 hybrid_clock.cc:648] HybridClock initialized: now 1790414892163324 us; error 30 us; skew 500 ppm
I20260926 09:28:12.164695 10425 webserver.cc:460] Webserver started at http://127.0.98.3:43199/ using document root <none> and password file <none>
I20260926 09:28:12.164959 10425 fs_manager.cc:359] Metadata directory not provided
I20260926 09:28:12.165030 10425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:28:12.165169 10425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:28:12.166294 10425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "80094c83521343fb9e87df88521969b2"
format_stamp: "Formatted at 2026-09-26 09:28:12 on dist-test-slave-dfzm"
server_key: "08b659d605c7d690aa7aef79623c72ff"
server_key_iv: "0fc57c57a77529f0972ff99fb5eff44d"
server_key_version: "encryptionkey@0"
I20260926 09:28:12.166651 10425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "80094c83521343fb9e87df88521969b2"
format_stamp: "Formatted at 2026-09-26 09:28:12 on dist-test-slave-dfzm"
server_key: "08b659d605c7d690aa7aef79623c72ff"
server_key_iv: "0fc57c57a77529f0972ff99fb5eff44d"
server_key_version: "encryptionkey@0"
I20260926 09:28:12.167943 10425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 09:28:12.168627 10439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:12.168792 10425 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:28:12.168895 10425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "80094c83521343fb9e87df88521969b2"
format_stamp: "Formatted at 2026-09-26 09:28:12 on dist-test-slave-dfzm"
server_key: "08b659d605c7d690aa7aef79623c72ff"
server_key_iv: "0fc57c57a77529f0972ff99fb5eff44d"
server_key_version: "encryptionkey@0"
I20260926 09:28:12.168970 10425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:28:12.172453 10425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:28:12.172745 10425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:28:12.172892 10425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:28:12.173105 10425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:28:12.173426 10425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:28:12.173482 10425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:12.173522 10425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:28:12.173549 10425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:12.179692 10425 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42579
I20260926 09:28:12.179728 10552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42579 every 8 connection(s)
I20260926 09:28:12.180060 10425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:28:12.183261 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:12.180749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41567 (local address 127.0.98.62:45243)
0926 09:28:12.180791 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:12.180793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:12.181069 (+ 276us) server_negotiation.cc:407] Connection header received
0926 09:28:12.181633 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:12.181634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:12.181640 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:12.181689 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:12.182952 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:28:12.182953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:28:12.182954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:28:12.182959 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:28:12.182973 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:28:12.182978 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:28:12.183040 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:12.183171 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 09:28:12.183187 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:28:12.183478 10555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:12.180745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45243 (local address 127.0.98.3:41567)
0926 09:28:12.180945 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:12.180980 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:12.181479 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:12.181772 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:12.181814 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:12.181826 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:12.182773 (+ 947us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:28:12.182773 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:12.182803 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:28:12.182832 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:28:12.183039 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:28:12.183043 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:28:12.183046 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:28:12.183182 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 09:28:12.183252 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260926 09:28:12.183988 10553 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45243
I20260926 09:28:12.184064 10553 heartbeater.cc:464] Registering TS with master...
I20260926 09:28:12.184285 10553 heartbeater.cc:511] Master 127.0.98.62:45243 requested a full tablet report, sending...
I20260926 09:28:12.184657 10103 ts_manager.cc:194] Registered new tserver with Master: 80094c83521343fb9e87df88521969b2 (127.0.98.3:42579)
I20260926 09:28:12.186442 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 10425
I20260926 09:28:12.186518 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:28:12.186729 392 external_mini_cluster.cc:1428] Setting key 229c73fc2fedfcba8050c553481658d5
I20260926 09:28:12.187587 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 09:28:12.189260 10155 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:28:12.188851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52550 (local address 127.0.98.62:45243)
0926 09:28:12.188892 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:12.188894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:12.188967 (+ 73us) server_negotiation.cc:407] Connection header received
0926 09:28:12.189021 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:12.189022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:12.189025 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:12.189075 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:12.189180 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52550: BlockingRecv error: recv got EOF from 127.0.0.1:52550 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:28:12.189370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 10163
I20260926 09:28:12.195022 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 10294
I20260926 09:28:12.199695 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 10425
I20260926 09:28:12.203991 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 10072
2026-09-26T09:28:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (553 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:28:12 dist-test-slave-dfzm krb5kdc[10563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:28:12 dist-test-slave-dfzm krb5kdc[10563](info): set up 2 sockets
Sep 26 09:28:12 dist-test-slave-dfzm krb5kdc[10563](info): commencing operation
krb5kdc: starting...
W20260926 09:28:13.250589 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:28:13 dist-test-slave-dfzm krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:28:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:28:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:28:13.372452 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33557
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:28:13.452710 10576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:28:13.452925 10576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:28:13.452948 10576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:28:13.454160 10576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:28:13.454210 10576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:28:13.454226 10576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:28:13.454237 10576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:28:13.454249 10576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:28:13.455739 10576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:28:13.456001 10576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:28:13.456231 10576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:28:13.461140 10581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:28:13.461280 10576 server_base.cc:1034] running on GCE node
W20260926 09:28:13.461158 10584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:28:13.461218 10582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:28:13.461722 10576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:28:13.462067 10576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:28:13.463213 10576 hybrid_clock.cc:648] HybridClock initialized: now 1790414893463194 us; error 37 us; skew 500 ppm
Sep 26 09:28:13 dist-test-slave-dfzm krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:28:13.465888 10576 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:28:13.466483 10576 webserver.cc:460] Webserver started at http://127.0.98.62:41359/ using document root <none> and password file <none>
I20260926 09:28:13.466758 10576 fs_manager.cc:359] Metadata directory not provided
I20260926 09:28:13.466813 10576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:28:13.466992 10576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:28:13.468201 10576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "782691b334224988bfdc429689cc63a4"
format_stamp: "Formatted at 2026-09-26 09:28:13 on dist-test-slave-dfzm"
server_key: "3579d786556037aeff5656ac17bb46fb"
server_key_iv: "8389ba41bbad7ad6fda7b6e7ed9bd3a8"
server_key_version: "encryptionkey@0"
I20260926 09:28:13.468623 10576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "782691b334224988bfdc429689cc63a4"
format_stamp: "Formatted at 2026-09-26 09:28:13 on dist-test-slave-dfzm"
server_key: "3579d786556037aeff5656ac17bb46fb"
server_key_iv: "8389ba41bbad7ad6fda7b6e7ed9bd3a8"
server_key_version: "encryptionkey@0"
I20260926 09:28:13.470239 10576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:28:13.471170 10591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:13.471359 10576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:28:13.471467 10576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "782691b334224988bfdc429689cc63a4"
format_stamp: "Formatted at 2026-09-26 09:28:13 on dist-test-slave-dfzm"
server_key: "3579d786556037aeff5656ac17bb46fb"
server_key_iv: "8389ba41bbad7ad6fda7b6e7ed9bd3a8"
server_key_version: "encryptionkey@0"
I20260926 09:28:13.471549 10576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:28:13.490535 10576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:28:13.492471 10576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:28:13.492636 10576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:28:13.497434 10576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33557
I20260926 09:28:13.497509 10643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33557 every 8 connection(s)
I20260926 09:28:13.497874 10576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:28:13.498845 10644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:28:13.501004 10644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 782691b334224988bfdc429689cc63a4: Bootstrap starting.
I20260926 09:28:13.501641 10644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 782691b334224988bfdc429689cc63a4: Neither blocks nor log segments found. Creating new log.
I20260926 09:28:13.501951 10644 log.cc:824] T 00000000000000000000000000000000 P 782691b334224988bfdc429689cc63a4: Log is configured to *not* fsync() on all Append() calls
I20260926 09:28:13.502689 10644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 782691b334224988bfdc429689cc63a4: No bootstrap required, opened a new log
I20260926 09:28:13.504204 10644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 782691b334224988bfdc429689cc63a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782691b334224988bfdc429689cc63a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33557 } }
I20260926 09:28:13.504364 10644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 782691b334224988bfdc429689cc63a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:28:13.504390 10644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 782691b334224988bfdc429689cc63a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782691b334224988bfdc429689cc63a4, State: Initialized, Role: FOLLOWER
I20260926 09:28:13.504495 10644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 782691b334224988bfdc429689cc63a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782691b334224988bfdc429689cc63a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33557 } }
I20260926 09:28:13.504575 10644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 782691b334224988bfdc429689cc63a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:28:13.504609 10644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 782691b334224988bfdc429689cc63a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:28:13.504663 10644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 782691b334224988bfdc429689cc63a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:28:13.505478 10644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 782691b334224988bfdc429689cc63a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782691b334224988bfdc429689cc63a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33557 } }
I20260926 09:28:13.505623 10644 leader_election.cc:302] T 00000000000000000000000000000000 P 782691b334224988bfdc429689cc63a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 782691b334224988bfdc429689cc63a4; no voters:
I20260926 09:28:13.505796 10644 leader_election.cc:288] T 00000000000000000000000000000000 P 782691b334224988bfdc429689cc63a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:28:13.505893 10647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 782691b334224988bfdc429689cc63a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:28:13.506057 10644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 782691b334224988bfdc429689cc63a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:28:13.506059 10647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 782691b334224988bfdc429689cc63a4 [term 1 LEADER]: Becoming Leader. State: Replica: 782691b334224988bfdc429689cc63a4, State: Running, Role: LEADER
I20260926 09:28:13.506246 10647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 782691b334224988bfdc429689cc63a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782691b334224988bfdc429689cc63a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33557 } }
I20260926 09:28:13.506649 10649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782691b334224988bfdc429689cc63a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 782691b334224988bfdc429689cc63a4. Latest consensus state: current_term: 1 leader_uuid: "782691b334224988bfdc429689cc63a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782691b334224988bfdc429689cc63a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33557 } } }
I20260926 09:28:13.506654 10648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782691b334224988bfdc429689cc63a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "782691b334224988bfdc429689cc63a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782691b334224988bfdc429689cc63a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33557 } } }
I20260926 09:28:13.506747 10649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782691b334224988bfdc429689cc63a4 [sys.catalog]: This master's current role is: LEADER
I20260926 09:28:13.506774 10648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782691b334224988bfdc429689cc63a4 [sys.catalog]: This master's current role is: LEADER
I20260926 09:28:13.506950 10656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:28:13.507449 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 10576
I20260926 09:28:13.507555 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:28:13.507718 10656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:28:13.507771 392 external_mini_cluster.cc:1428] Setting key 1f53fdac7f4a1d84d57c7c863d916cd1
I20260926 09:28:13.509539 10656 catalog_manager.cc:1347] Generated new cluster ID: b6737ff3050f4356ba284687d74de002
I20260926 09:28:13.509618 10656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:28:13 dist-test-slave-dfzm krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:28:13.519940 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:13.508280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48438 (local address 127.0.98.62:33557)
0926 09:28:13.508874 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:13.508881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:13.508911 (+ 30us) server_negotiation.cc:407] Connection header received
0926 09:28:13.509054 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:13.509063 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:13.509151 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:13.509312 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:13.510098 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:13.511518 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:13.512561 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:13.513217 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:13.513648 (+ 431us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:13.516238 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:28:13.516243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:28:13.516243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:28:13.516256 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:28:13.518255 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:28:13.518665 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:28:13.518667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:28:13.518668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:28:13.518767 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:28:13.518974 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:28:13.518975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:28:13.518975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:28:13.519281 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:28:13.519398 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:13.519555 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 09:28:13.519693 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":87,"threads_started":1}
I20260926 09:28:13.540630 10656 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:28:13.541178 10656 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:28:13.547776 10656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 782691b334224988bfdc429689cc63a4: Generated new TSK 0
I20260926 09:28:13.548007 10656 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:28:13.582685 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33557
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:28:13.698604 10670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:28:13.698846 10670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:28:13.698877 10670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:28:13.700201 10670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:28:13.700253 10670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:28:13.700316 10670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:28:13.701826 10670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:28:13.702236 10670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:28:13.702515 10670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:28:13.707796 10676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:28:13.707928 10670 server_base.cc:1034] running on GCE node
W20260926 09:28:13.707870 10678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:28:13.708048 10675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:28:13.708364 10670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:28:13.708588 10670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:28:13.709764 10670 hybrid_clock.cc:648] HybridClock initialized: now 1790414893709717 us; error 40 us; skew 500 ppm
Sep 26 09:28:13 dist-test-slave-dfzm krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:28:13.712510 10670 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:28:13.712932 10670 webserver.cc:460] Webserver started at http://127.0.98.1:42613/ using document root <none> and password file <none>
I20260926 09:28:13.713178 10670 fs_manager.cc:359] Metadata directory not provided
I20260926 09:28:13.713253 10670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:28:13.713398 10670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:28:13.714556 10670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "5938b1ec6e9b4d5eb31ce0682396791c"
format_stamp: "Formatted at 2026-09-26 09:28:13 on dist-test-slave-dfzm"
server_key: "617395f1d749487c020a1be9731b423c"
server_key_iv: "4df9b698adf4e2ea0a5bedc033ddde50"
server_key_version: "encryptionkey@0"
I20260926 09:28:13.714962 10670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5938b1ec6e9b4d5eb31ce0682396791c"
format_stamp: "Formatted at 2026-09-26 09:28:13 on dist-test-slave-dfzm"
server_key: "617395f1d749487c020a1be9731b423c"
server_key_iv: "4df9b698adf4e2ea0a5bedc033ddde50"
server_key_version: "encryptionkey@0"
I20260926 09:28:13.716346 10670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:28:13.717090 10685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:13.717260 10670 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:28:13.717363 10670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "5938b1ec6e9b4d5eb31ce0682396791c"
format_stamp: "Formatted at 2026-09-26 09:28:13 on dist-test-slave-dfzm"
server_key: "617395f1d749487c020a1be9731b423c"
server_key_iv: "4df9b698adf4e2ea0a5bedc033ddde50"
server_key_version: "encryptionkey@0"
I20260926 09:28:13.717453 10670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:28:13.732483 10670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:28:13.735244 10670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:28:13.735435 10670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:28:13.735708 10670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:28:13.736058 10670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:28:13.736119 10670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:13.736160 10670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:28:13.736213 10670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:13.742784 10670 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44371
I20260926 09:28:13.742825 10798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44371 every 8 connection(s)
I20260926 09:28:13.743240 10670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:28:13.748482 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 10670
I20260926 09:28:13.748601 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:28:13.748821 392 external_mini_cluster.cc:1428] Setting key 4b59bfdbfd636256282031c359316816
Sep 26 09:28:13 dist-test-slave-dfzm krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:28:13.753656 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:13.744047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53001 (local address 127.0.98.62:33557)
0926 09:28:13.744138 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:13.744140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:13.744362 (+ 222us) server_negotiation.cc:407] Connection header received
0926 09:28:13.744800 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:13.744801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:13.744814 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:13.744885 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:13.745971 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:13.746502 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:13.747524 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:13.747727 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:13.747790 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:13.750430 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:28:13.750435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:28:13.750435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:28:13.750444 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:28:13.752071 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:28:13.752541 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:28:13.752543 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:28:13.752543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:28:13.752616 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:28:13.752906 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:28:13.752907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:28:13.752907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:28:13.753037 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:28:13.753121 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:13.753420 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 09:28:13.753520 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 09:28:13.753856 10801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:13.744000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33557 (local address 127.0.98.1:53001)
0926 09:28:13.744179 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:13.744221 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:13.744708 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:13.744960 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:13.744964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:13.745567 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:13.745845 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:13.745850 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:13.746591 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:13.746591 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:13.747434 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:13.747436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:13.747807 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:13.747808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:13.747938 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:13.748526 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:13.748537 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:13.750313 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:28:13.752171 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:28:13.752179 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:28:13.752181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:28:13.752435 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:28:13.752703 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:28:13.752704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:28:13.752704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:28:13.752829 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:28:13.753110 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:28:13.753117 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:28:13.753276 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 09:28:13.753397 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 09:28:13.753566 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
I20260926 09:28:13.754359 10799 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33557
I20260926 09:28:13.754473 10799 heartbeater.cc:464] Registering TS with master...
I20260926 09:28:13.754679 10799 heartbeater.cc:511] Master 127.0.98.62:33557 requested a full tablet report, sending...
I20260926 09:28:13.755304 10608 ts_manager.cc:194] Registered new tserver with Master: 5938b1ec6e9b4d5eb31ce0682396791c (127.0.98.1:44371)
I20260926 09:28:13.756259 10608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53001
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:28:13.774163 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33557
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:28:13.857699 10804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:28:13.857928 10804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:28:13.857952 10804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:28:13.859146 10804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:28:13.859186 10804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:28:13.859212 10804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:28:13.860626 10804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:28:13.860805 10804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:28:13.860977 10804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:28:13.865965 10812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:28:13.866060 10809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:28:13.866009 10810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:28:13.866192 10804 server_base.cc:1034] running on GCE node
I20260926 09:28:13.866415 10804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:28:13.866654 10804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:28:13.867832 10804 hybrid_clock.cc:648] HybridClock initialized: now 1790414893867818 us; error 31 us; skew 500 ppm
Sep 26 09:28:13 dist-test-slave-dfzm krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:28:13.870625 10804 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:28:13.871133 10804 webserver.cc:460] Webserver started at http://127.0.98.2:45581/ using document root <none> and password file <none>
I20260926 09:28:13.871399 10804 fs_manager.cc:359] Metadata directory not provided
I20260926 09:28:13.871471 10804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:28:13.871614 10804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:28:13.872648 10804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "6194a5ccf8af47259db6c813d6f14f99"
format_stamp: "Formatted at 2026-09-26 09:28:13 on dist-test-slave-dfzm"
server_key: "7993553a2621972efd1b7efa927169e7"
server_key_iv: "13b523bd89b952d84c9ca1fec40147f9"
server_key_version: "encryptionkey@0"
I20260926 09:28:13.873018 10804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6194a5ccf8af47259db6c813d6f14f99"
format_stamp: "Formatted at 2026-09-26 09:28:13 on dist-test-slave-dfzm"
server_key: "7993553a2621972efd1b7efa927169e7"
server_key_iv: "13b523bd89b952d84c9ca1fec40147f9"
server_key_version: "encryptionkey@0"
I20260926 09:28:13.874693 10804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:28:13.875543 10819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:13.875727 10804 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:28:13.875828 10804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "6194a5ccf8af47259db6c813d6f14f99"
format_stamp: "Formatted at 2026-09-26 09:28:13 on dist-test-slave-dfzm"
server_key: "7993553a2621972efd1b7efa927169e7"
server_key_iv: "13b523bd89b952d84c9ca1fec40147f9"
server_key_version: "encryptionkey@0"
I20260926 09:28:13.875916 10804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:28:13.890921 10804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:28:13.892891 10804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:28:13.893062 10804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:28:13.893302 10804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:28:13.893677 10804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:28:13.893733 10804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:13.893772 10804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:28:13.893800 10804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:13.899706 10804 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41733
I20260926 09:28:13.899749 10932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41733 every 8 connection(s)
I20260926 09:28:13.900111 10804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:28:13 dist-test-slave-dfzm krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:28:13.909332 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 10804
I20260926 09:28:13.909458 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:28:13.909713 392 external_mini_cluster.cc:1428] Setting key 53b97f100c0bbd04d73154d0b85b43cd
I20260926 09:28:13.911386 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:13.900864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33557 (local address 127.0.98.62:33557)
0926 09:28:13.900918 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:13.900921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:13.901145 (+ 224us) server_negotiation.cc:407] Connection header received
0926 09:28:13.901654 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:13.901655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:13.901666 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:13.901718 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:13.902743 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:13.903240 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:13.904372 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:13.904559 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:13.904615 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:13.907361 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:28:13.907363 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:28:13.907364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:28:13.907373 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:28:13.909492 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:28:13.910105 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:28:13.910106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:28:13.910106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:28:13.910175 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:28:13.910559 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:28:13.910560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:28:13.910561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:28:13.910711 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:28:13.910782 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:13.911178 (+ 396us) server_negotiation.cc:299] Negotiation successful
0926 09:28:13.911275 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:28:13.911950 10935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:13.900851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33557 (local address 127.0.98.2:33557)
0926 09:28:13.901023 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:13.901058 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:13.901544 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:13.901769 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:13.901773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:13.902454 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:13.902648 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:13.902652 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:13.903356 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:13.903357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:13.904284 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:13.904286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:13.904635 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:13.904635 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:13.904731 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:13.905282 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:13.905292 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:13.907252 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:28:13.909622 (+ 2370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:28:13.909629 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:28:13.909632 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:28:13.909993 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:28:13.910280 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:28:13.910281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:28:13.910281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:28:13.910450 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:28:13.910822 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:28:13.910828 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:28:13.911004 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 09:28:13.911487 (+ 483us) client_negotiation.cc:240] Negotiation successful
0926 09:28:13.911686 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260926 09:28:13.912433 10933 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33557
I20260926 09:28:13.912539 10933 heartbeater.cc:464] Registering TS with master...
I20260926 09:28:13.912725 10933 heartbeater.cc:511] Master 127.0.98.62:33557 requested a full tablet report, sending...
I20260926 09:28:13.913141 10608 ts_manager.cc:194] Registered new tserver with Master: 6194a5ccf8af47259db6c813d6f14f99 (127.0.98.2:41733)
I20260926 09:28:13.913745 10608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33557
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:28:13.936332 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33557
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:28:14.017311 10938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:28:14.017584 10938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:28:14.017644 10938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:28:14.018813 10938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:28:14.018887 10938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:28:14.018934 10938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:28:14.020568 10938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:28:14.020848 10938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:28:14.021095 10938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:28:14.026412 10943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:28:14.026467 10944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:28:14.026502 10938 server_base.cc:1034] running on GCE node
W20260926 09:28:14.026434 10946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:28:14.026754 10938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:28:14.026973 10938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:28:14.028142 10938 hybrid_clock.cc:648] HybridClock initialized: now 1790414894028125 us; error 29 us; skew 500 ppm
Sep 26 09:28:14 dist-test-slave-dfzm krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:28:14.030658 10938 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:28:14.031100 10938 webserver.cc:460] Webserver started at http://127.0.98.3:39259/ using document root <none> and password file <none>
I20260926 09:28:14.031297 10938 fs_manager.cc:359] Metadata directory not provided
I20260926 09:28:14.031337 10938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:28:14.031432 10938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:28:14.032449 10938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "6cf66ae4a91a44869e130250572d08bf"
format_stamp: "Formatted at 2026-09-26 09:28:14 on dist-test-slave-dfzm"
server_key: "0018d22d4d966346b16ec851da23128b"
server_key_iv: "2db898faf89846b6b4a9d19a31072d78"
server_key_version: "encryptionkey@0"
I20260926 09:28:14.032738 10938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6cf66ae4a91a44869e130250572d08bf"
format_stamp: "Formatted at 2026-09-26 09:28:14 on dist-test-slave-dfzm"
server_key: "0018d22d4d966346b16ec851da23128b"
server_key_iv: "2db898faf89846b6b4a9d19a31072d78"
server_key_version: "encryptionkey@0"
I20260926 09:28:14.034466 10938 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 09:28:14.035526 10953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:14.035727 10938 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:28:14.035841 10938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "6cf66ae4a91a44869e130250572d08bf"
format_stamp: "Formatted at 2026-09-26 09:28:14 on dist-test-slave-dfzm"
server_key: "0018d22d4d966346b16ec851da23128b"
server_key_iv: "2db898faf89846b6b4a9d19a31072d78"
server_key_version: "encryptionkey@0"
I20260926 09:28:14.035923 10938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:28:14.056859 10938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:28:14.059015 10938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:28:14.059171 10938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:28:14.059408 10938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:28:14.059723 10938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:28:14.059757 10938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:14.059808 10938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:28:14.059836 10938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:14.065727 10938 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43479
I20260926 09:28:14.065771 11066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43479 every 8 connection(s)
I20260926 09:28:14.066105 10938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:28:14.071394 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 10938
I20260926 09:28:14.071485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:28:14.071671 392 external_mini_cluster.cc:1428] Setting key 2a32f80767bc496c9b44e27bf00938a1
Sep 26 09:28:14 dist-test-slave-dfzm krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:28:14.075626 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:14.066823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37649 (local address 127.0.98.62:33557)
0926 09:28:14.066896 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:14.066898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:14.067115 (+ 217us) server_negotiation.cc:407] Connection header received
0926 09:28:14.067663 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:14.067664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:14.067674 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:14.067735 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:14.068652 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.069134 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.070232 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.070421 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.070479 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:14.072822 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:28:14.072823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:28:14.072823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:28:14.072829 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:28:14.074409 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:28:14.074813 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:28:14.074814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:28:14.074814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:28:14.074873 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:28:14.075101 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:28:14.075101 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:28:14.075102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:28:14.075203 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:28:14.075256 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:14.075496 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 09:28:14.075562 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 09:28:14.075865 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:14.066807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33557 (local address 127.0.98.3:37649)
0926 09:28:14.066979 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:14.067022 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:14.067574 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:14.067805 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:14.067808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:14.068386 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:14.068574 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:14.068578 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.069215 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.069216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:14.070154 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:14.070156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.070499 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.070500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:14.070600 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:14.071067 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:14.071076 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:14.072743 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:28:14.074505 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:28:14.074511 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:28:14.074513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:28:14.074749 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:28:14.074935 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:28:14.074935 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:28:14.074936 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:28:14.075046 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:28:14.075262 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:28:14.075266 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:28:14.075382 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 09:28:14.075464 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 09:28:14.075625 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
I20260926 09:28:14.076364 11067 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33557
I20260926 09:28:14.076445 11067 heartbeater.cc:464] Registering TS with master...
I20260926 09:28:14.076638 11067 heartbeater.cc:511] Master 127.0.98.62:33557 requested a full tablet report, sending...
I20260926 09:28:14.077033 10608 ts_manager.cc:194] Registered new tserver with Master: 6cf66ae4a91a44869e130250572d08bf (127.0.98.3:43479)
I20260926 09:28:14.077423 10608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37649
I20260926 09:28:14.083576 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:28:14.090651 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:14.084850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48452 (local address 127.0.98.62:33557)
0926 09:28:14.084918 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:14.084920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:14.085042 (+ 122us) server_negotiation.cc:407] Connection header received
0926 09:28:14.085085 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:14.085086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:14.085095 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:14.085145 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:14.085735 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.086212 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.086920 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.087094 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.087143 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:14.088265 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:28:14.088266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:28:14.088266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:28:14.088272 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:28:14.089742 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:28:14.090038 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:28:14.090039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:28:14.090040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:28:14.090089 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:28:14.090246 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:28:14.090246 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:28:14.090247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:28:14.090380 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:28:14.090451 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:14.090537 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 09:28:14.090589 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 09:28:14.092226 10608 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:28:14.093827 10608 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:28:14.100394 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:14.096333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42898 (local address 127.0.98.3:43479)
0926 09:28:14.096723 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:14.096727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:14.096736 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:28:14.096774 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:14.096776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:14.096792 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:14.096871 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:28:14.097083 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.098212 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.099458 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.099950 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.100041 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:14.100097 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:14.100250 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 09:28:14.100295 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":104,"threads_started":1}
I20260926 09:28:14.100394 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:14.096162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43479 (local address 127.0.0.1:42898)
0926 09:28:14.096442 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:14.096485 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:14.096556 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:14.096954 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:14.096955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:14.096957 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:28:14.097003 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:14.097005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.098337 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.098338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:14.099367 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:14.099368 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.100040 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.100041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:14.100118 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:14.100131 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 09:28:14.100233 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 09:28:14.100300 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":27,"thread_start_us":93,"threads_started":1}
I20260926 09:28:14.100553 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:14.096455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58360 (local address 127.0.98.2:41733)
0926 09:28:14.096649 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:14.096653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:14.096715 (+ 62us) server_negotiation.cc:407] Connection header received
0926 09:28:14.096832 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:14.096834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:14.096849 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:14.096926 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:28:14.097114 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.098237 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.099372 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.100053 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.100111 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:14.100156 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:14.100443 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 09:28:14.100481 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":90,"threads_started":1}
I20260926 09:28:14.100559 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:14.096366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46562 (local address 127.0.98.1:44371)
0926 09:28:14.096554 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:14.096560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:14.096574 (+ 14us) server_negotiation.cc:407] Connection header received
0926 09:28:14.096660 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:14.096663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:14.096683 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:14.096758 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:28:14.097003 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.097815 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.099211 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.099958 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.100031 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:14.100094 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:14.100443 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 09:28:14.100485 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":83,"threads_started":1}
I20260926 09:28:14.100404 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:14.096162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44371 (local address 127.0.0.1:46562)
0926 09:28:14.096520 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:14.096531 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:14.096585 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:14.096849 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:14.096852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:14.096854 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:28:14.096923 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:14.096925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.097914 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.097915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:14.099126 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:14.099129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.100130 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.100131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:14.100197 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:14.100200 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:28:14.100249 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 09:28:14.100304 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":5,"thread_start_us":59,"threads_started":1}
I20260926 09:28:14.100404 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:14.096176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41733 (local address 127.0.0.1:58360)
0926 09:28:14.096657 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:14.096666 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:14.096718 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:14.096999 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:14.097000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:14.097001 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:28:14.097041 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:14.097042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.098340 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.098341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:14.099279 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:14.099282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.100184 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.100184 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:14.100243 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:14.100247 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:28:14.100291 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 09:28:14.100320 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":100,"threads_started":1}
I20260926 09:28:14.101434 10733 tablet_service.cc:1467] Processing CreateTablet for tablet cf719275a4a046cca1199cf3c001d236 (DEFAULT_TABLE table=test-table [id=04d0d326d49748fca7759ca054d396e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:28:14.101516 11001 tablet_service.cc:1467] Processing CreateTablet for tablet cf719275a4a046cca1199cf3c001d236 (DEFAULT_TABLE table=test-table [id=04d0d326d49748fca7759ca054d396e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:28:14.101827 10733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf719275a4a046cca1199cf3c001d236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:28:14.101827 11001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf719275a4a046cca1199cf3c001d236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:28:14.102043 10867 tablet_service.cc:1467] Processing CreateTablet for tablet cf719275a4a046cca1199cf3c001d236 (DEFAULT_TABLE table=test-table [id=04d0d326d49748fca7759ca054d396e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:28:14.102384 10867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf719275a4a046cca1199cf3c001d236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:28:14.105037 11084 tablet_bootstrap.cc:492] T cf719275a4a046cca1199cf3c001d236 P 6194a5ccf8af47259db6c813d6f14f99: Bootstrap starting.
I20260926 09:28:14.105355 11086 tablet_bootstrap.cc:492] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c: Bootstrap starting.
I20260926 09:28:14.105762 11084 tablet_bootstrap.cc:654] T cf719275a4a046cca1199cf3c001d236 P 6194a5ccf8af47259db6c813d6f14f99: Neither blocks nor log segments found. Creating new log.
I20260926 09:28:14.105752 11085 tablet_bootstrap.cc:492] T cf719275a4a046cca1199cf3c001d236 P 6cf66ae4a91a44869e130250572d08bf: Bootstrap starting.
I20260926 09:28:14.106144 11086 tablet_bootstrap.cc:654] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c: Neither blocks nor log segments found. Creating new log.
I20260926 09:28:14.106305 11084 log.cc:824] T cf719275a4a046cca1199cf3c001d236 P 6194a5ccf8af47259db6c813d6f14f99: Log is configured to *not* fsync() on all Append() calls
I20260926 09:28:14.106405 11085 tablet_bootstrap.cc:654] T cf719275a4a046cca1199cf3c001d236 P 6cf66ae4a91a44869e130250572d08bf: Neither blocks nor log segments found. Creating new log.
I20260926 09:28:14.106822 11086 log.cc:824] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c: Log is configured to *not* fsync() on all Append() calls
I20260926 09:28:14.106969 11085 log.cc:824] T cf719275a4a046cca1199cf3c001d236 P 6cf66ae4a91a44869e130250572d08bf: Log is configured to *not* fsync() on all Append() calls
I20260926 09:28:14.107093 11084 tablet_bootstrap.cc:492] T cf719275a4a046cca1199cf3c001d236 P 6194a5ccf8af47259db6c813d6f14f99: No bootstrap required, opened a new log
I20260926 09:28:14.107169 11084 ts_tablet_manager.cc:1397] T cf719275a4a046cca1199cf3c001d236 P 6194a5ccf8af47259db6c813d6f14f99: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:28:14.107671 11086 tablet_bootstrap.cc:492] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c: No bootstrap required, opened a new log
I20260926 09:28:14.107748 11085 tablet_bootstrap.cc:492] T cf719275a4a046cca1199cf3c001d236 P 6cf66ae4a91a44869e130250572d08bf: No bootstrap required, opened a new log
I20260926 09:28:14.107759 11086 ts_tablet_manager.cc:1397] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:28:14.107813 11085 ts_tablet_manager.cc:1397] T cf719275a4a046cca1199cf3c001d236 P 6cf66ae4a91a44869e130250572d08bf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:28:14.109242 11086 raft_consensus.cc:348] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938b1ec6e9b4d5eb31ce0682396791c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } } peers { permanent_uuid: "6cf66ae4a91a44869e130250572d08bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43479 } } peers { permanent_uuid: "6194a5ccf8af47259db6c813d6f14f99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41733 } }
I20260926 09:28:14.109205 11084 raft_consensus.cc:348] T cf719275a4a046cca1199cf3c001d236 P 6194a5ccf8af47259db6c813d6f14f99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938b1ec6e9b4d5eb31ce0682396791c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } } peers { permanent_uuid: "6cf66ae4a91a44869e130250572d08bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43479 } } peers { permanent_uuid: "6194a5ccf8af47259db6c813d6f14f99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41733 } }
I20260926 09:28:14.109392 11084 raft_consensus.cc:374] T cf719275a4a046cca1199cf3c001d236 P 6194a5ccf8af47259db6c813d6f14f99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:28:14.109413 11086 raft_consensus.cc:374] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:28:14.109431 11084 raft_consensus.cc:729] T cf719275a4a046cca1199cf3c001d236 P 6194a5ccf8af47259db6c813d6f14f99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6194a5ccf8af47259db6c813d6f14f99, State: Initialized, Role: FOLLOWER
I20260926 09:28:14.109459 11086 raft_consensus.cc:729] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5938b1ec6e9b4d5eb31ce0682396791c, State: Initialized, Role: FOLLOWER
I20260926 09:28:14.109570 11086 consensus_queue.cc:260] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938b1ec6e9b4d5eb31ce0682396791c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } } peers { permanent_uuid: "6cf66ae4a91a44869e130250572d08bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43479 } } peers { permanent_uuid: "6194a5ccf8af47259db6c813d6f14f99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41733 } }
I20260926 09:28:14.109570 11084 consensus_queue.cc:260] T cf719275a4a046cca1199cf3c001d236 P 6194a5ccf8af47259db6c813d6f14f99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938b1ec6e9b4d5eb31ce0682396791c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } } peers { permanent_uuid: "6cf66ae4a91a44869e130250572d08bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43479 } } peers { permanent_uuid: "6194a5ccf8af47259db6c813d6f14f99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41733 } }
I20260926 09:28:14.109877 11086 ts_tablet_manager.cc:1428] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 09:28:14.109897 10799 heartbeater.cc:503] Master 127.0.98.62:33557 was elected leader, sending a full tablet report...
I20260926 09:28:14.109802 11085 raft_consensus.cc:348] T cf719275a4a046cca1199cf3c001d236 P 6cf66ae4a91a44869e130250572d08bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938b1ec6e9b4d5eb31ce0682396791c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } } peers { permanent_uuid: "6cf66ae4a91a44869e130250572d08bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43479 } } peers { permanent_uuid: "6194a5ccf8af47259db6c813d6f14f99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41733 } }
I20260926 09:28:14.109949 11085 raft_consensus.cc:374] T cf719275a4a046cca1199cf3c001d236 P 6cf66ae4a91a44869e130250572d08bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:28:14.109995 11085 raft_consensus.cc:729] T cf719275a4a046cca1199cf3c001d236 P 6cf66ae4a91a44869e130250572d08bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf66ae4a91a44869e130250572d08bf, State: Initialized, Role: FOLLOWER
I20260926 09:28:14.110054 10933 heartbeater.cc:503] Master 127.0.98.62:33557 was elected leader, sending a full tablet report...
I20260926 09:28:14.110097 11085 consensus_queue.cc:260] T cf719275a4a046cca1199cf3c001d236 P 6cf66ae4a91a44869e130250572d08bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938b1ec6e9b4d5eb31ce0682396791c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } } peers { permanent_uuid: "6cf66ae4a91a44869e130250572d08bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43479 } } peers { permanent_uuid: "6194a5ccf8af47259db6c813d6f14f99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41733 } }
I20260926 09:28:14.110177 11084 ts_tablet_manager.cc:1428] T cf719275a4a046cca1199cf3c001d236 P 6194a5ccf8af47259db6c813d6f14f99: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260926 09:28:14.110354 11085 ts_tablet_manager.cc:1428] T cf719275a4a046cca1199cf3c001d236 P 6cf66ae4a91a44869e130250572d08bf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 09:28:14.110468 11067 heartbeater.cc:503] Master 127.0.98.62:33557 was elected leader, sending a full tablet report...
W20260926 09:28:14.150671 10934 tablet.cc:2369] T cf719275a4a046cca1199cf3c001d236 P 6194a5ccf8af47259db6c813d6f14f99: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:28:14.243969 10800 tablet.cc:2369] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:28:14.316702 11068 tablet.cc:2369] T cf719275a4a046cca1199cf3c001d236 P 6cf66ae4a91a44869e130250572d08bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:28:14.418210 11090 raft_consensus.cc:482] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:28:14.418411 11090 raft_consensus.cc:504] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938b1ec6e9b4d5eb31ce0682396791c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } } peers { permanent_uuid: "6cf66ae4a91a44869e130250572d08bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43479 } } peers { permanent_uuid: "6194a5ccf8af47259db6c813d6f14f99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41733 } }
I20260926 09:28:14.418851 11090 leader_election.cc:288] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cf66ae4a91a44869e130250572d08bf (127.0.98.3:43479), 6194a5ccf8af47259db6c813d6f14f99 (127.0.98.2:41733)
I20260926 09:28:14.423134 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:14.419090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43479 (local address 127.0.98.1:48887)
0926 09:28:14.419260 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:14.419303 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:14.419381 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:14.419563 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:14.419564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:14.419568 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:28:14.419664 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:14.419666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.420643 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.420644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:14.422218 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:14.422223 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.422819 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.422820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:14.422884 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:14.422889 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:28:14.422995 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 09:28:14.423043 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":82,"thread_start_us":81,"threads_started":1}
I20260926 09:28:14.423118 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:14.419176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48887 (local address 127.0.98.3:43479)
0926 09:28:14.419224 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:14.419226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:14.419383 (+ 157us) server_negotiation.cc:407] Connection header received
0926 09:28:14.419470 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:14.419471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:14.419482 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:14.419538 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:28:14.419743 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.420526 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.422329 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.422703 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.422768 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:14.422777 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:14.423007 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 09:28:14.423051 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:28:14.423664 11021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf719275a4a046cca1199cf3c001d236" candidate_uuid: "5938b1ec6e9b4d5eb31ce0682396791c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf66ae4a91a44869e130250572d08bf" is_pre_election: true
I20260926 09:28:14.423852 11021 raft_consensus.cc:2454] T cf719275a4a046cca1199cf3c001d236 P 6cf66ae4a91a44869e130250572d08bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5938b1ec6e9b4d5eb31ce0682396791c in term 0.
I20260926 09:28:14.424172 10687 leader_election.cc:302] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5938b1ec6e9b4d5eb31ce0682396791c, 6cf66ae4a91a44869e130250572d08bf; no voters:
I20260926 09:28:14.424271 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:14.419211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36301 (local address 127.0.98.2:41733)
0926 09:28:14.419292 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:14.419294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:14.419383 (+ 89us) server_negotiation.cc:407] Connection header received
0926 09:28:14.419446 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:14.419447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:14.419463 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:14.419533 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:28:14.419743 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.420511 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.422842 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.423329 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.423398 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:14.423410 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:14.424150 (+ 740us) server_negotiation.cc:299] Negotiation successful
0926 09:28:14.424204 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 09:28:14.424294 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:14.419102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41733 (local address 127.0.98.1:36301)
0926 09:28:14.419277 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:14.419288 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:14.419381 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:14.419563 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:14.419564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:14.419566 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:28:14.419658 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:14.419660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.420642 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.420643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:14.422242 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:14.422244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.423995 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.423996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:14.424066 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:14.424070 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:28:14.424140 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 09:28:14.424175 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":127,"thread_start_us":66,"threads_started":1}
I20260926 09:28:14.424336 11090 raft_consensus.cc:2781] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:28:14.424386 11090 raft_consensus.cc:482] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:28:14.424414 11090 raft_consensus.cc:3037] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:28:14.424621 10887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf719275a4a046cca1199cf3c001d236" candidate_uuid: "5938b1ec6e9b4d5eb31ce0682396791c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6194a5ccf8af47259db6c813d6f14f99" is_pre_election: true
I20260926 09:28:14.424744 10887 raft_consensus.cc:2454] T cf719275a4a046cca1199cf3c001d236 P 6194a5ccf8af47259db6c813d6f14f99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5938b1ec6e9b4d5eb31ce0682396791c in term 0.
I20260926 09:28:14.425269 11090 raft_consensus.cc:504] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938b1ec6e9b4d5eb31ce0682396791c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } } peers { permanent_uuid: "6cf66ae4a91a44869e130250572d08bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43479 } } peers { permanent_uuid: "6194a5ccf8af47259db6c813d6f14f99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41733 } }
I20260926 09:28:14.425427 11090 leader_election.cc:288] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c [CANDIDATE]: Term 1 election: Requested vote from peers 6cf66ae4a91a44869e130250572d08bf (127.0.98.3:43479), 6194a5ccf8af47259db6c813d6f14f99 (127.0.98.2:41733)
I20260926 09:28:14.425666 11021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf719275a4a046cca1199cf3c001d236" candidate_uuid: "5938b1ec6e9b4d5eb31ce0682396791c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf66ae4a91a44869e130250572d08bf"
I20260926 09:28:14.425668 10887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf719275a4a046cca1199cf3c001d236" candidate_uuid: "5938b1ec6e9b4d5eb31ce0682396791c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6194a5ccf8af47259db6c813d6f14f99"
I20260926 09:28:14.425743 11021 raft_consensus.cc:3037] T cf719275a4a046cca1199cf3c001d236 P 6cf66ae4a91a44869e130250572d08bf [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:28:14.425782 10887 raft_consensus.cc:3037] T cf719275a4a046cca1199cf3c001d236 P 6194a5ccf8af47259db6c813d6f14f99 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:28:14.426534 11021 raft_consensus.cc:2454] T cf719275a4a046cca1199cf3c001d236 P 6cf66ae4a91a44869e130250572d08bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5938b1ec6e9b4d5eb31ce0682396791c in term 1.
I20260926 09:28:14.426558 10887 raft_consensus.cc:2454] T cf719275a4a046cca1199cf3c001d236 P 6194a5ccf8af47259db6c813d6f14f99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5938b1ec6e9b4d5eb31ce0682396791c in term 1.
I20260926 09:28:14.426708 10687 leader_election.cc:302] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5938b1ec6e9b4d5eb31ce0682396791c, 6cf66ae4a91a44869e130250572d08bf; no voters:
I20260926 09:28:14.426817 11090 raft_consensus.cc:2781] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:28:14.427024 11090 raft_consensus.cc:686] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c [term 1 LEADER]: Becoming Leader. State: Replica: 5938b1ec6e9b4d5eb31ce0682396791c, State: Running, Role: LEADER
I20260926 09:28:14.427129 11090 consensus_queue.cc:237] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938b1ec6e9b4d5eb31ce0682396791c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } } peers { permanent_uuid: "6cf66ae4a91a44869e130250572d08bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43479 } } peers { permanent_uuid: "6194a5ccf8af47259db6c813d6f14f99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41733 } }
I20260926 09:28:14.427980 10607 catalog_manager.cc:5495] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5938b1ec6e9b4d5eb31ce0682396791c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5938b1ec6e9b4d5eb31ce0682396791c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938b1ec6e9b4d5eb31ce0682396791c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cf66ae4a91a44869e130250572d08bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6194a5ccf8af47259db6c813d6f14f99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41733 } health_report { overall_health: UNKNOWN } } }
I20260926 09:28:14.457077 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:14.454667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35464 (local address 127.0.98.1:44371)
0926 09:28:14.454745 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:14.454748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:14.454759 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:28:14.454825 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:14.454826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:14.454846 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:14.454928 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:28:14.455095 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.455593 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.456325 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:14.456502 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:14.456555 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:14.456721 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:28:14.456827 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:28:14.456899 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:14.456938 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 09:28:14.456955 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 09:28:14.459028 10887 raft_consensus.cc:1261] T cf719275a4a046cca1199cf3c001d236 P 6194a5ccf8af47259db6c813d6f14f99 [term 1 FOLLOWER]: Refusing update from remote peer 5938b1ec6e9b4d5eb31ce0682396791c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:28:14.459029 11021 raft_consensus.cc:1261] T cf719275a4a046cca1199cf3c001d236 P 6cf66ae4a91a44869e130250572d08bf [term 1 FOLLOWER]: Refusing update from remote peer 5938b1ec6e9b4d5eb31ce0682396791c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:28:14.459462 11095 consensus_queue.cc:1035] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6194a5ccf8af47259db6c813d6f14f99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:28:14.459600 11090 consensus_queue.cc:1035] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cf66ae4a91a44869e130250572d08bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:28:14.462162 11100 mvcc.cc:205] Tried to move back new op lower bound from 7333539407700779008 to 7333539407573987328. Current Snapshot: MvccSnapshot[applied={T|T < 7333539407700779008}]
I20260926 09:28:14.464475 10608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:28:14.464567 10608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:28:14.465338 10608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 782691b334224988bfdc429689cc63a4: Sending DeleteTablet for 3 replicas of tablet cf719275a4a046cca1199cf3c001d236
I20260926 09:28:14.465678 10733 tablet_service.cc:1514] Processing DeleteTablet for tablet cf719275a4a046cca1199cf3c001d236 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:28:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46562
I20260926 09:28:14.465704 11001 tablet_service.cc:1514] Processing DeleteTablet for tablet cf719275a4a046cca1199cf3c001d236 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:28:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42898
I20260926 09:28:14.465895 10867 tablet_service.cc:1514] Processing DeleteTablet for tablet cf719275a4a046cca1199cf3c001d236 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:28:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58360
I20260926 09:28:14.466107 11109 tablet_replica.cc:330] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c: stopping tablet replica
I20260926 09:28:14.466208 11109 raft_consensus.cc:2229] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:28:14.466264 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 10670
I20260926 09:28:14.466217 11110 tablet_replica.cc:330] T cf719275a4a046cca1199cf3c001d236 P 6194a5ccf8af47259db6c813d6f14f99: stopping tablet replica
I20260926 09:28:14.466337 11109 raft_consensus.cc:2258] T cf719275a4a046cca1199cf3c001d236 P 5938b1ec6e9b4d5eb31ce0682396791c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:28:14.466450 11108 tablet_replica.cc:330] T cf719275a4a046cca1199cf3c001d236 P 6cf66ae4a91a44869e130250572d08bf: stopping tablet replica
I20260926 09:28:14.466548 11108 raft_consensus.cc:2229] T cf719275a4a046cca1199cf3c001d236 P 6cf66ae4a91a44869e130250572d08bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:28:14.466459 11110 raft_consensus.cc:2229] T cf719275a4a046cca1199cf3c001d236 P 6194a5ccf8af47259db6c813d6f14f99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:28:14.466724 11110 raft_consensus.cc:2258] T cf719275a4a046cca1199cf3c001d236 P 6194a5ccf8af47259db6c813d6f14f99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:28:14.466640 11108 raft_consensus.cc:2258] T cf719275a4a046cca1199cf3c001d236 P 6cf66ae4a91a44869e130250572d08bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:28:14.467478 11108 ts_tablet_manager.cc:1905] T cf719275a4a046cca1199cf3c001d236 P 6cf66ae4a91a44869e130250572d08bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:28:14.468261 11110 ts_tablet_manager.cc:1905] T cf719275a4a046cca1199cf3c001d236 P 6194a5ccf8af47259db6c813d6f14f99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:28:14.469354 11108 ts_tablet_manager.cc:1918] T cf719275a4a046cca1199cf3c001d236 P 6cf66ae4a91a44869e130250572d08bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:28:14.469442 11108 log.cc:1196] T cf719275a4a046cca1199cf3c001d236 P 6cf66ae4a91a44869e130250572d08bf: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/cf719275a4a046cca1199cf3c001d236
I20260926 09:28:14.469743 11108 ts_tablet_manager.cc:1939] T cf719275a4a046cca1199cf3c001d236 P 6cf66ae4a91a44869e130250572d08bf: Deleting consensus metadata
I20260926 09:28:14.470180 10593 catalog_manager.cc:4841] TS 6cf66ae4a91a44869e130250572d08bf (127.0.98.3:43479): tablet cf719275a4a046cca1199cf3c001d236 (table test-table [id=04d0d326d49748fca7759ca054d396e6]) successfully deleted
I20260926 09:28:14.470268 11110 ts_tablet_manager.cc:1918] T cf719275a4a046cca1199cf3c001d236 P 6194a5ccf8af47259db6c813d6f14f99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:28:14.470345 11110 log.cc:1196] T cf719275a4a046cca1199cf3c001d236 P 6194a5ccf8af47259db6c813d6f14f99: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414839069479-392-0/minicluster-data/ts-1/wal/wals/cf719275a4a046cca1199cf3c001d236
I20260926 09:28:14.470590 11110 ts_tablet_manager.cc:1939] T cf719275a4a046cca1199cf3c001d236 P 6194a5ccf8af47259db6c813d6f14f99: Deleting consensus metadata
I20260926 09:28:14.471120 10594 catalog_manager.cc:4841] TS 6194a5ccf8af47259db6c813d6f14f99 (127.0.98.2:41733): tablet cf719275a4a046cca1199cf3c001d236 (table test-table [id=04d0d326d49748fca7759ca054d396e6]) successfully deleted
W20260926 09:28:14.472401 10592 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44371 (error 108)
I20260926 09:28:14.472934 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 10804
I20260926 09:28:14.472970 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:14.472824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44371 (local address 127.0.0.1:35474)
0926 09:28:14.472887 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:14.472941 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 09:28:14.477746 10592 catalog_manager.cc:4584] TS 5938b1ec6e9b4d5eb31ce0682396791c (127.0.98.1:44371): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cf719275a4a046cca1199cf3c001d236: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44371: connect: Connection refused (error 111)
I20260926 09:28:14.478039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 10938
I20260926 09:28:14.482409 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 10576
2026-09-26T09:28:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2279 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:28:14 dist-test-slave-dfzm krb5kdc[11114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:28:14 dist-test-slave-dfzm krb5kdc[11114](info): set up 2 sockets
Sep 26 09:28:14 dist-test-slave-dfzm krb5kdc[11114](info): commencing operation
krb5kdc: starting...
W20260926 09:28:15.523183 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:28:15 dist-test-slave-dfzm krb5kdc[11114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:28:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:28:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 09:28:15.684197 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskNX8gJ6/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/postgres -l logfile start
2026-09-26 09:28:18.195 UTC [11144] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 09:28:18.195 UTC [11144] LOG: listening on IPv4 address "127.0.98.20", port 41791
2026-09-26 09:28:18.199 UTC [11144] LOG: listening on Unix socket "/tmp/.s.PGSQL.41791"
2026-09-26 09:28:18.221 UTC [11147] LOG: database system was shut down at 2026-09-26 09:28:16 UTC
2026-09-26 09:28:18.224 UTC [11144] LOG: database system is ready to accept connections
I20260926 09:28:19.175103 392 mini_postgres.cc:96] Postgres bound to 41791
2026-09-26 09:28:19.181 UTC [11155] FATAL: database "slave" does not exist
127.0.98.20:41791 - accepting connections
I20260926 09:28:19.181905 392 mini_ranger.cc:161] Starting Ranger...
I20260926 09:28:19.196966 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 09:28:19.686599 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 09:28:19.686694 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ranger-admin
2026-09-26 09:28:19,965 [I] DB FLAVOR :POSTGRES
2026-09-26 09:28:19,965 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 09:28:19,965 [I] Checking connection..
2026-09-26 09:28:19,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 09:28:20,196 [I] Checking connection passed.
2026-09-26 09:28:20,196 [I] --------- Verifying version history table ---------
2026-09-26 09:28:20,196 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 09:28:20,432 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 09:28:20,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 09:28:20,693 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 09:28:20,693 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 09:28:20,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41791/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 09:28:20.938 UTC [11219] WARNING: there is no transaction in progress
2026-09-26 09:28:20,944 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 09:28:20,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 09:28:21,180 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 09:28:21,181 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 09:28:21,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 09:28:21,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 09:28:21,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dfzm.c.gcp-upstream.internal','N') ;"
2026-09-26 09:28:21,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 09:28:22,174 [I] Table x_portal_user does not exist in database ranger
2026-09-26 09:28:22,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 09:28:22,409 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 09:28:22,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 09:28:22,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 09:28:22,890 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 09:28:22,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41791/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 09:28:23.847 UTC [11327] WARNING: there is no transaction in progress
2026-09-26 09:28:23.864 UTC [11327] WARNING: there is no transaction in progress
2026-09-26 09:28:23.881 UTC [11327] WARNING: there is no transaction in progress
2026-09-26 09:28:23.896 UTC [11327] WARNING: there is no transaction in progress
2026-09-26 09:28:23.912 UTC [11327] WARNING: there is no transaction in progress
2026-09-26 09:28:23.933 UTC [11327] WARNING: there is no transaction in progress
2026-09-26 09:28:23.997 UTC [11327] WARNING: there is no transaction in progress
2026-09-26 09:28:24.003 UTC [11327] WARNING: there is no transaction in progress
2026-09-26 09:28:24.017 UTC [11327] WARNING: there is no transaction in progress
2026-09-26 09:28:24.028 UTC [11327] WARNING: there is no transaction in progress
2026-09-26 09:28:24.038 UTC [11327] WARNING: there is no transaction in progress
2026-09-26 09:28:24.051 UTC [11327] WARNING: there is no transaction in progress
2026-09-26 09:28:24.061 UTC [11327] WARNING: there is no transaction in progress
2026-09-26 09:28:24.068 UTC [11327] WARNING: there is no transaction in progress
2026-09-26 09:28:24.074 UTC [11327] WARNING: there is no transaction in progress
2026-09-26 09:28:24.080 UTC [11327] WARNING: there is no transaction in progress
2026-09-26 09:28:24,721 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 09:28:24,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 09:28:24,944 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 09:28:24,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 09:28:25,204 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 09:28:25,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 09:28:25,460 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 09:28:25,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 09:28:25,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 09:28:25,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dfzm.c.gcp-upstream.internal';"
2026-09-26 09:28:26,238 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 09:28:26,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 09:28:26,470 [I] DB_PATCHES have already been applied
I20260926 09:28:26.476274 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskNX8gJ6/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 09:28:26.476366 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 09:28:26.479297 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:55831
Sep 26, 2026 9:28:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 9:28:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 9:28:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 9:28:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 9:28:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 9:28:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 9:28:27 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 09:28:27.579517 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.398s user 0.000s sys 0.009s
Sep 26, 2026 9:28:46 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 9:28:46 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 26, 2026 9:28:46 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 9:28:46 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 9:28:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 9:28:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 9:28:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 9:28:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 9:28:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 9:28:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 9:28:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 9:28:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 9:28:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 9:28:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 9:28:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 9:28:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 9:28:46 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 9:28:47 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 09:28:48.475466 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 09:28:48 dist-test-slave-dfzm krb5kdc[11114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414928, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 09:28:48.562772 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 09:28:48.572670 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 09:28:49.440316 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 09:28:49.440392 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ranger-kms
I20260926 09:28:49.440404 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:41791
2026-09-26 09:28:49,625 [I] DB FLAVOR :POSTGRES
2026-09-26 09:28:49,626 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 09:28:49,626 [I] Checking connection
2026-09-26 09:28:49,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41791/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 09:28:49,896 [I] connection success
2026-09-26 09:28:49,897 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 09:28:49,897 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 09:28:49,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41791/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 09:28:50,144 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 09:28:50,144 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 09:28:50,144 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 09:28:50,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41791/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 09:28:50,408 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 09:28:50.849161 392 mini_ranger_kms.cc:310] Created kms service
I20260926 09:28:51.040869 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 09:28:51.083413 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 09:28:51 dist-test-slave-dfzm krb5kdc[11114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414928, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:28:51.464082 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 09:28:51.464203 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 09:28:51.464226 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 09:28:51.467525 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11507
I20260926 09:28:51.467651 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:58751
Sep 26, 2026 9:28:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 9:28:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 9:28:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 9:28:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 9:28:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 9:28:53 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 09:28:53.799712 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.237s user 0.000s sys 0.155s
I20260926 09:28:53.799822 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 09:28:53.799894 392 mini_ranger_kms.cc:401] 127.0.98.20:58751/kms/v1/keys
Sep 26 09:28:53 dist-test-slave-dfzm krb5kdc[11114](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790414933, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 09:28:54 dist-test-slave-dfzm krb5kdc[11114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414933, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 09:28:54 dist-test-slave-dfzm krb5kdc[11114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414933, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 09:28:57 dist-test-slave-dfzm krb5kdc[11114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:28:57.169064 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42241
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42241
--ranger_config_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58751/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:28:57.256528 11558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:28:57.256742 11558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:28:57.256764 11558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:28:57.257983 11558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:28:57.258025 11558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:28:57.258040 11558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:28:57.258054 11558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:28:57.258100 11558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:28:57.259634 11558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:58751/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42241
--ranger_config_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:28:57.259871 11558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:28:57.260115 11558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:28:57.265848 11563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:28:57.265843 11564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:28:57.265859 11558 server_base.cc:1034] running on GCE node
W20260926 09:28:57.265843 11566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:28:57.266330 11558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:28:57.266638 11558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:28:57.267858 11558 hybrid_clock.cc:648] HybridClock initialized: now 1790414937267836 us; error 38 us; skew 500 ppm
Sep 26 09:28:57 dist-test-slave-dfzm krb5kdc[11114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:28:57.275449 11558 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:28:57.275961 11558 webserver.cc:460] Webserver started at http://127.0.98.62:34677/ using document root <none> and password file <none>
I20260926 09:28:57.276206 11558 fs_manager.cc:359] Metadata directory not provided
I20260926 09:28:57.276265 11558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:28:57.276425 11558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 09:28:57 dist-test-slave-dfzm krb5kdc[11114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:28:57.323210 11558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "d164169f5f6546e2a26bc6d66244c305"
format_stamp: "Formatted at 2026-09-26 09:28:57 on dist-test-slave-dfzm"
server_key: "32fe05ae56661fcc91038393993316d6"
server_key_iv: "3250584a793bd574bcf2eb038e5bf259"
server_key_version: "kuduclusterkey@0"
I20260926 09:28:57.323624 11558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "d164169f5f6546e2a26bc6d66244c305"
format_stamp: "Formatted at 2026-09-26 09:28:57 on dist-test-slave-dfzm"
server_key: "32fe05ae56661fcc91038393993316d6"
server_key_iv: "3250584a793bd574bcf2eb038e5bf259"
server_key_version: "kuduclusterkey@0"
I20260926 09:28:57.325100 11558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:28:57.337049 11573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:57.337273 11558 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:28:57.337375 11558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "d164169f5f6546e2a26bc6d66244c305"
format_stamp: "Formatted at 2026-09-26 09:28:57 on dist-test-slave-dfzm"
server_key: "32fe05ae56661fcc91038393993316d6"
server_key_iv: "3250584a793bd574bcf2eb038e5bf259"
server_key_version: "kuduclusterkey@0"
I20260926 09:28:57.337495 11558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:28:57.352469 11558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:28:57.354611 11558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:28:57.354809 11558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:28:57.360042 11558 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42241
I20260926 09:28:57.360057 11625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42241 every 8 connection(s)
I20260926 09:28:57.360463 11558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:28:57.361270 11626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:28:57.364356 11626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d164169f5f6546e2a26bc6d66244c305: Bootstrap starting.
I20260926 09:28:57.365061 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 11558
I20260926 09:28:57.365094 11626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d164169f5f6546e2a26bc6d66244c305: Neither blocks nor log segments found. Creating new log.
I20260926 09:28:57.365175 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:28:57.365357 392 external_mini_cluster.cc:1428] Setting key 18d42f847c4c35e6bb29a9b9b3193cfc
I20260926 09:28:57.365409 11626 log.cc:824] T 00000000000000000000000000000000 P d164169f5f6546e2a26bc6d66244c305: Log is configured to *not* fsync() on all Append() calls
I20260926 09:28:57.366287 11626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d164169f5f6546e2a26bc6d66244c305: No bootstrap required, opened a new log
I20260926 09:28:57.368270 11626 raft_consensus.cc:348] T 00000000000000000000000000000000 P d164169f5f6546e2a26bc6d66244c305 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d164169f5f6546e2a26bc6d66244c305" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42241 } }
I20260926 09:28:57.368432 11626 raft_consensus.cc:374] T 00000000000000000000000000000000 P d164169f5f6546e2a26bc6d66244c305 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:28:57.368495 11626 raft_consensus.cc:729] T 00000000000000000000000000000000 P d164169f5f6546e2a26bc6d66244c305 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d164169f5f6546e2a26bc6d66244c305, State: Initialized, Role: FOLLOWER
I20260926 09:28:57.368678 11626 consensus_queue.cc:260] T 00000000000000000000000000000000 P d164169f5f6546e2a26bc6d66244c305 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d164169f5f6546e2a26bc6d66244c305" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42241 } }
I20260926 09:28:57.368773 11626 raft_consensus.cc:388] T 00000000000000000000000000000000 P d164169f5f6546e2a26bc6d66244c305 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:28:57.368826 11626 raft_consensus.cc:482] T 00000000000000000000000000000000 P d164169f5f6546e2a26bc6d66244c305 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:28:57.368883 11626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d164169f5f6546e2a26bc6d66244c305 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:28:57.369418 11626 raft_consensus.cc:504] T 00000000000000000000000000000000 P d164169f5f6546e2a26bc6d66244c305 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d164169f5f6546e2a26bc6d66244c305" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42241 } }
I20260926 09:28:57.369544 11626 leader_election.cc:302] T 00000000000000000000000000000000 P d164169f5f6546e2a26bc6d66244c305 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d164169f5f6546e2a26bc6d66244c305; no voters:
I20260926 09:28:57.369755 11626 leader_election.cc:288] T 00000000000000000000000000000000 P d164169f5f6546e2a26bc6d66244c305 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:28:57.369804 11631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d164169f5f6546e2a26bc6d66244c305 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:28:57.370048 11631 raft_consensus.cc:686] T 00000000000000000000000000000000 P d164169f5f6546e2a26bc6d66244c305 [term 1 LEADER]: Becoming Leader. State: Replica: d164169f5f6546e2a26bc6d66244c305, State: Running, Role: LEADER
I20260926 09:28:57.370082 11626 sys_catalog.cc:564] T 00000000000000000000000000000000 P d164169f5f6546e2a26bc6d66244c305 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:28:57.370201 11631 consensus_queue.cc:237] T 00000000000000000000000000000000 P d164169f5f6546e2a26bc6d66244c305 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d164169f5f6546e2a26bc6d66244c305" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42241 } }
I20260926 09:28:57.370522 11631 sys_catalog.cc:455] T 00000000000000000000000000000000 P d164169f5f6546e2a26bc6d66244c305 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d164169f5f6546e2a26bc6d66244c305. Latest consensus state: current_term: 1 leader_uuid: "d164169f5f6546e2a26bc6d66244c305" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d164169f5f6546e2a26bc6d66244c305" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42241 } } }
I20260926 09:28:57.370576 11631 sys_catalog.cc:458] T 00000000000000000000000000000000 P d164169f5f6546e2a26bc6d66244c305 [sys.catalog]: This master's current role is: LEADER
I20260926 09:28:57.371702 11626 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 26 09:28:57 dist-test-slave-dfzm krb5kdc[11114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:28:57.376217 11632 sys_catalog.cc:455] T 00000000000000000000000000000000 P d164169f5f6546e2a26bc6d66244c305 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d164169f5f6546e2a26bc6d66244c305" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d164169f5f6546e2a26bc6d66244c305" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42241 } } }
I20260926 09:28:57.376413 11632 sys_catalog.cc:458] T 00000000000000000000000000000000 P d164169f5f6546e2a26bc6d66244c305 [sys.catalog]: This master's current role is: LEADER
I20260926 09:28:57.378547 11636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:28:57.379158 11636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:28:57.380697 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:57.365962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41562 (local address 127.0.98.62:42241)
0926 09:28:57.366141 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:57.366151 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:57.366182 (+ 31us) server_negotiation.cc:407] Connection header received
0926 09:28:57.366757 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:57.366766 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:57.366831 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:57.366981 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:57.367741 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:57.369655 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:57.370885 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:57.371525 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:57.371644 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:57.375383 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:28:57.375414 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:28:57.375414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:28:57.375453 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:28:57.378420 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:28:57.378879 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:28:57.378883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:28:57.378883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:28:57.378997 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:28:57.379212 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:28:57.379213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:28:57.379213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:28:57.379596 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:28:57.379763 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:57.380244 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 09:28:57.380423 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":60,"threads_started":1}
I20260926 09:28:57.386850 11636 catalog_manager.cc:1347] Generated new cluster ID: 962f0e7739c644aba877d21625607dad
I20260926 09:28:57.386893 11636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:28:57.393615 11636 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:28:57.394428 11636 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:28:57.401371 11636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d164169f5f6546e2a26bc6d66244c305: Generated new TSK 0
I20260926 09:28:57.401687 11636 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:28:58.109678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58751/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42241
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:28:58.227465 11671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:28:58.227694 11671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:28:58.227717 11671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:28:58.228925 11671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:28:58.228974 11671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:28:58.229004 11671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:28:58.230611 11671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:58751/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:28:58.230849 11671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:28:58.231140 11671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:28:58.236348 11677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:28:58.236495 11679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:28:58.236352 11676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:28:58.237033 11671 server_base.cc:1034] running on GCE node
I20260926 09:28:58.237280 11671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:28:58.237545 11671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:28:58.238775 11671 hybrid_clock.cc:648] HybridClock initialized: now 1790414938238748 us; error 44 us; skew 500 ppm
Sep 26 09:28:58 dist-test-slave-dfzm krb5kdc[11114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:28:58.241946 11671 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:28:58.242480 11671 webserver.cc:460] Webserver started at http://127.0.98.1:36629/ using document root <none> and password file <none>
I20260926 09:28:58.242731 11671 fs_manager.cc:359] Metadata directory not provided
I20260926 09:28:58.242799 11671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:28:58.242940 11671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 09:28:58 dist-test-slave-dfzm krb5kdc[11114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:28:58.257496 11671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "5c7b691b2150464fad0c9bc0a2b4c232"
format_stamp: "Formatted at 2026-09-26 09:28:58 on dist-test-slave-dfzm"
server_key: "379617156c2932618cf4c0b5631f67e5"
server_key_iv: "fe82f95aa1447164807a24cf29155921"
server_key_version: "kuduclusterkey@0"
I20260926 09:28:58.258080 11671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5c7b691b2150464fad0c9bc0a2b4c232"
format_stamp: "Formatted at 2026-09-26 09:28:58 on dist-test-slave-dfzm"
server_key: "379617156c2932618cf4c0b5631f67e5"
server_key_iv: "fe82f95aa1447164807a24cf29155921"
server_key_version: "kuduclusterkey@0"
I20260926 09:28:58.262020 11671 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260926 09:28:58.275293 11687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:58.275607 11671 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:28:58.275730 11671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "5c7b691b2150464fad0c9bc0a2b4c232"
format_stamp: "Formatted at 2026-09-26 09:28:58 on dist-test-slave-dfzm"
server_key: "379617156c2932618cf4c0b5631f67e5"
server_key_iv: "fe82f95aa1447164807a24cf29155921"
server_key_version: "kuduclusterkey@0"
I20260926 09:28:58.275837 11671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:28:58.295701 11671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:28:58.298424 11671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:28:58.298623 11671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:28:58.298911 11671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:28:58.299314 11671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:28:58.299405 11671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:58.299499 11671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:28:58.299571 11671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:58.307148 11671 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38721
I20260926 09:28:58.307646 11671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:28:58.309687 11801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38721 every 8 connection(s)
I20260926 09:28:58.316491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 11671
I20260926 09:28:58.316617 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:28:58.316835 392 external_mini_cluster.cc:1428] Setting key 1dbc3d3f4603184ba6deea9f49354dcf
Sep 26 09:28:58 dist-test-slave-dfzm krb5kdc[11114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:28:58.322621 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:58.310540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42241 (local address 127.0.98.1:42831)
0926 09:28:58.311142 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:58.311193 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:58.311816 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:58.312350 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:58.312354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:58.312600 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:58.312896 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:58.312902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.314063 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.314065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:58.315352 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:58.315355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.315902 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.315903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:58.316049 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:58.316762 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:58.316773 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:58.318606 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:28:58.320767 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:28:58.320775 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:28:58.320777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:28:58.321113 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:28:58.321353 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:28:58.321354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:28:58.321354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:28:58.321501 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:28:58.321838 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:28:58.321845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:28:58.322041 (+ 196us) client_negotiation.cc:769] Sending connection context
0926 09:28:58.322189 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 09:28:58.322363 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":101,"threads_started":1}
I20260926 09:28:58.323001 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:58.310866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42831 (local address 127.0.98.62:42241)
0926 09:28:58.311079 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:58.311081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:58.311978 (+ 897us) server_negotiation.cc:407] Connection header received
0926 09:28:58.312104 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:58.312104 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:58.312118 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:58.312214 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:58.313020 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.313967 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.315459 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.315757 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.315828 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:58.318723 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:28:58.318725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:28:58.318726 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:28:58.318734 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:28:58.320659 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:28:58.321214 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:28:58.321216 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:28:58.321216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:28:58.321284 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:28:58.321577 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:28:58.321578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:28:58.321578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:28:58.321746 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:28:58.321809 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:58.322795 (+ 986us) server_negotiation.cc:299] Negotiation successful
0926 09:28:58.322890 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":81,"threads_started":1}
I20260926 09:28:58.323537 11802 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42241
I20260926 09:28:58.323676 11802 heartbeater.cc:464] Registering TS with master...
I20260926 09:28:58.323884 11802 heartbeater.cc:511] Master 127.0.98.62:42241 requested a full tablet report, sending...
I20260926 09:28:58.324470 11590 ts_manager.cc:194] Registered new tserver with Master: 5c7b691b2150464fad0c9bc0a2b4c232 (127.0.98.1:38721)
I20260926 09:28:58.325382 11590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42831
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:28:58.357755 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58751/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42241
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:28:58 dist-test-slave-dfzm krb5kdc[11114](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790414938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 09:28:58.494870 11808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:28:58.495180 11808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:28:58.495220 11808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:28:58.497110 11808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:28:58.497174 11808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:28:58.497215 11808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:28:58.499461 11808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:58751/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:28:58.499866 11808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:28:58.500185 11808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:28:58.506592 11816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:28:58.506805 11818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:28:58.506971 11808 server_base.cc:1034] running on GCE node
W20260926 09:28:58.507292 11815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:28:58.507550 11808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:28:58.507836 11808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:28:58.509012 11808 hybrid_clock.cc:648] HybridClock initialized: now 1790414938508992 us; error 36 us; skew 500 ppm
Sep 26 09:28:58 dist-test-slave-dfzm krb5kdc[11114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:28:58.512285 11808 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:28:58.512863 11808 webserver.cc:460] Webserver started at http://127.0.98.2:39013/ using document root <none> and password file <none>
I20260926 09:28:58.513120 11808 fs_manager.cc:359] Metadata directory not provided
I20260926 09:28:58.513183 11808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:28:58.513331 11808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 09:28:58 dist-test-slave-dfzm krb5kdc[11114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:28:58.529897 11808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "70287eea7b9a4d0a9bb9cd40b6a99b4f"
format_stamp: "Formatted at 2026-09-26 09:28:58 on dist-test-slave-dfzm"
server_key: "8fde0cc1413297155afca203baff0201"
server_key_iv: "6c6872be1de63b1955709ee0a91203e7"
server_key_version: "kuduclusterkey@0"
I20260926 09:28:58.530426 11808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "70287eea7b9a4d0a9bb9cd40b6a99b4f"
format_stamp: "Formatted at 2026-09-26 09:28:58 on dist-test-slave-dfzm"
server_key: "8fde0cc1413297155afca203baff0201"
server_key_iv: "6c6872be1de63b1955709ee0a91203e7"
server_key_version: "kuduclusterkey@0"
I20260926 09:28:58.533798 11808 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 09:28:58.558470 11826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:58.558797 11808 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.000s
I20260926 09:28:58.558944 11808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "70287eea7b9a4d0a9bb9cd40b6a99b4f"
format_stamp: "Formatted at 2026-09-26 09:28:58 on dist-test-slave-dfzm"
server_key: "8fde0cc1413297155afca203baff0201"
server_key_iv: "6c6872be1de63b1955709ee0a91203e7"
server_key_version: "kuduclusterkey@0"
I20260926 09:28:58.559070 11808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:28:58.576292 11808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:28:58.578934 11808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:28:58.579156 11808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:28:58.579500 11808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:28:58.579995 11808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:28:58.580090 11808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:58.580273 11808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:28:58.580345 11808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:58.588027 11808 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38973
I20260926 09:28:58.588560 11808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:28:58.589293 11939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38973 every 8 connection(s)
I20260926 09:28:58.590464 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 11808
I20260926 09:28:58.590575 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:28:58.590778 392 external_mini_cluster.cc:1428] Setting key a5f426eb6b18bd3f70d6882990d5282b
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 26 09:28:58 dist-test-slave-dfzm krb5kdc[11114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 09:28:58.604872 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:58.591071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37683 (local address 127.0.98.62:42241)
0926 09:28:58.591126 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:58.591128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:58.593884 (+ 2756us) server_negotiation.cc:407] Connection header received
0926 09:28:58.594576 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:58.594577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:58.594591 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:58.594664 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:58.595520 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.596278 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.597853 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.598151 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.598202 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:58.601273 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:28:58.601275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:28:58.601275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:28:58.601284 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:28:58.603182 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:28:58.603731 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:28:58.603732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:28:58.603733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:28:58.603792 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:28:58.604077 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:28:58.604078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:28:58.604078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:28:58.604214 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:28:58.604273 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:58.604675 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 09:28:58.604768 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 09:28:58.605415 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:58.590070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42241 (local address 127.0.98.2:37683)
0926 09:28:58.593724 (+ 3654us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:58.593789 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:58.594448 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:58.594809 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:58.594814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:58.595081 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:58.595397 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:58.595404 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.596409 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.596410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:58.597746 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:58.597749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.598320 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.598321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:58.598472 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:58.599545 (+ 1073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:58.599557 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:58.601158 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:28:58.603268 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:28:58.603275 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:28:58.603278 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:28:58.603630 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:28:58.603865 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:28:58.603865 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:28:58.603866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:28:58.604000 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:28:58.604321 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:28:58.604328 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:28:58.604517 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 09:28:58.605004 (+ 487us) client_negotiation.cc:240] Negotiation successful
0926 09:28:58.605171 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3597,"thread_start_us":97,"threads_started":1}
I20260926 09:28:58.606103 11940 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42241
I20260926 09:28:58.606213 11940 heartbeater.cc:464] Registering TS with master...
I20260926 09:28:58.606415 11940 heartbeater.cc:511] Master 127.0.98.62:42241 requested a full tablet report, sending...
I20260926 09:28:58.606819 11590 ts_manager.cc:194] Registered new tserver with Master: 70287eea7b9a4d0a9bb9cd40b6a99b4f (127.0.98.2:38973)
I20260926 09:28:58.607312 11590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:28:58.624987 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58751/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42241
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:28:58.760926 11945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:28:58.761322 11945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:28:58.761420 11945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:28:58.763585 11945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:28:58.763715 11945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:28:58.763840 11945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:28:58.766551 11945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:58751/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:28:58.767045 11945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:28:58.767405 11945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:28:58.776063 11951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:28:58.776088 11953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:28:58.776682 11950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:28:58.777736 11945 server_base.cc:1034] running on GCE node
I20260926 09:28:58.777998 11945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:28:58.778302 11945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:28:58.793788 11945 hybrid_clock.cc:648] HybridClock initialized: now 1790414938793742 us; error 48 us; skew 500 ppm
Sep 26 09:28:58 dist-test-slave-dfzm krb5kdc[11114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:28:58.799017 11945 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:28:58.799752 11945 webserver.cc:460] Webserver started at http://127.0.98.3:33337/ using document root <none> and password file <none>
I20260926 09:28:58.800112 11945 fs_manager.cc:359] Metadata directory not provided
I20260926 09:28:58.800246 11945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:28:58.800431 11945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 09:28:58 dist-test-slave-dfzm krb5kdc[11114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:28:58.823240 11945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "47e468cad6544156bb934e944f5e20e7"
format_stamp: "Formatted at 2026-09-26 09:28:58 on dist-test-slave-dfzm"
server_key: "4d0a07752f949878cc180ba2adcc8a3e"
server_key_iv: "d2c70387f377b18b1445bd74b6637a1e"
server_key_version: "kuduclusterkey@0"
I20260926 09:28:58.823917 11945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "47e468cad6544156bb934e944f5e20e7"
format_stamp: "Formatted at 2026-09-26 09:28:58 on dist-test-slave-dfzm"
server_key: "4d0a07752f949878cc180ba2adcc8a3e"
server_key_iv: "d2c70387f377b18b1445bd74b6637a1e"
server_key_version: "kuduclusterkey@0"
I20260926 09:28:58.825965 11945 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 09:28:58.856175 11960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:58.856442 11945 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260926 09:28:58.856534 11945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "47e468cad6544156bb934e944f5e20e7"
format_stamp: "Formatted at 2026-09-26 09:28:58 on dist-test-slave-dfzm"
server_key: "4d0a07752f949878cc180ba2adcc8a3e"
server_key_iv: "d2c70387f377b18b1445bd74b6637a1e"
server_key_version: "kuduclusterkey@0"
I20260926 09:28:58.856686 11945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:28:58.871982 11945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:28:58.874536 11945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:28:58.874812 11945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:28:58.875092 11945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:28:58.875509 11945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:28:58.875610 11945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:58.875695 11945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:28:58.875759 11945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:28:58.882885 11945 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41169
I20260926 09:28:58.883442 11945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:28:58.887082 12073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41169 every 8 connection(s)
I20260926 09:28:58.890574 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 11945
I20260926 09:28:58.890702 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:28:58.890900 392 external_mini_cluster.cc:1428] Setting key 67202d5f05beb252e632218887e6a014
Sep 26 09:28:58 dist-test-slave-dfzm krb5kdc[11114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:28:58.908206 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:58.887638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42241 (local address 127.0.98.3:43357)
0926 09:28:58.888034 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:58.888078 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:58.888717 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:58.889065 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:58.889069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:58.889320 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:28:58.889629 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:58.889635 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.890632 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.890633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:58.892644 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:58.892648 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.893307 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.893308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:58.893470 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:58.901669 (+ 8199us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:28:58.901685 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:28:58.904154 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:28:58.906435 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:28:58.906445 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:28:58.906448 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:28:58.906794 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:28:58.907013 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:28:58.907013 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:28:58.907014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:28:58.907159 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:28:58.907465 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:28:58.907472 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:28:58.907682 (+ 210us) client_negotiation.cc:769] Sending connection context
0926 09:28:58.907796 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 09:28:58.907961 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":82,"threads_started":1}
I20260926 09:28:58.908746 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:58.887918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43357 (local address 127.0.98.62:42241)
0926 09:28:58.887963 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:58.887965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:58.888171 (+ 206us) server_negotiation.cc:407] Connection header received
0926 09:28:58.888841 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:58.888842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:58.888854 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:58.888884 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:58.889738 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.890509 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.892767 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.893144 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.893192 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:58.904284 (+ 11092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:28:58.904286 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:28:58.904286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:28:58.904295 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:28:58.906324 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:28:58.906884 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:28:58.906885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:28:58.906885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:28:58.906945 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:28:58.907231 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:28:58.907232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:28:58.907232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:28:58.907375 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:28:58.907434 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:58.908559 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0926 09:28:58.908654 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:28:58.909441 12074 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42241
I20260926 09:28:58.909538 12074 heartbeater.cc:464] Registering TS with master...
I20260926 09:28:58.909762 12074 heartbeater.cc:511] Master 127.0.98.62:42241 requested a full tablet report, sending...
I20260926 09:28:58.910120 11590 ts_manager.cc:194] Registered new tserver with Master: 47e468cad6544156bb934e944f5e20e7 (127.0.98.3:41169)
I20260926 09:28:58.910629 11590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43357
I20260926 09:28:58.913115 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:28:58.942922 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:58.921917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41570 (local address 127.0.98.62:42241)
0926 09:28:58.921971 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:58.921973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:58.933830 (+ 11857us) server_negotiation.cc:407] Connection header received
0926 09:28:58.933972 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:58.933974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:58.933988 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:58.934026 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:28:58.934955 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.935828 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.937104 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.937408 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.937461 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:58.939335 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:28:58.939337 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:28:58.939337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:28:58.939346 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:28:58.941407 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:28:58.941905 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:28:58.941907 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:28:58.941907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:28:58.941984 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:28:58.942223 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:28:58.942224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:28:58.942225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:28:58.942368 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:28:58.942465 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:58.942720 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 09:28:58.942812 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:28:58.945465 11590 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:28:58.947952 11590 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:28:58.960824 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:58.953023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59546 (local address 127.0.98.2:38973)
0926 09:28:58.953851 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:58.953857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:58.954025 (+ 168us) server_negotiation.cc:407] Connection header received
0926 09:28:58.954073 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:58.954076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:58.954095 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:58.954169 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:28:58.954437 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.956029 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.958797 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.959718 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.959800 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:58.959891 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:58.960674 (+ 783us) server_negotiation.cc:299] Negotiation successful
0926 09:28:58.960722 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":788,"thread_start_us":95,"threads_started":1}
I20260926 09:28:58.961220 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:58.952562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38973 (local address 127.0.0.1:59546)
0926 09:28:58.953903 (+ 1341us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:58.953913 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:58.953962 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:58.954269 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:58.954272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:58.954274 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:28:58.954346 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:58.954347 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.956140 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.956142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:58.958672 (+ 2530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:58.958674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.959931 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.959932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:58.960030 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:58.960067 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 09:28:58.960165 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 09:28:58.960232 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1326,"thread_start_us":82,"threads_started":1}
I20260926 09:28:58.962435 11863 tablet_service.cc:1467] Processing CreateTablet for tablet 7a9714552cea4764a76baac2147078b0 (DEFAULT_TABLE table=test-table [id=b7f2e6d2d3d8454eb55daa1d774ed3f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:28:58.962857 11863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a9714552cea4764a76baac2147078b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:28:58.968815 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:58.952746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41169 (local address 127.0.0.1:33312)
0926 09:28:58.957865 (+ 5119us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:58.957877 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:58.958386 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:58.962257 (+ 3871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:58.962259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:58.962261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:28:58.962321 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:58.962323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.963656 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.963657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:58.965249 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:58.965252 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.966459 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.966460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:58.966536 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:58.966541 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:28:58.968667 (+ 2126us) client_negotiation.cc:240] Negotiation successful
0926 09:28:58.968721 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5100,"thread_start_us":85,"threads_started":1}
I20260926 09:28:58.972182 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:58.952322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38721 (local address 127.0.0.1:60444)
0926 09:28:58.953544 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:58.953610 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:58.953690 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:58.966613 (+ 12923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:58.966613 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:58.966630 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:28:58.966685 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:58.966686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.969249 (+ 2563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.969250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:58.970675 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:58.970678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.971898 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.971899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:58.972008 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:58.972013 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:28:58.972047 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 09:28:58.972090 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1187,"thread_start_us":88,"threads_started":1}
I20260926 09:28:58.972821 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:58.952413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60444 (local address 127.0.98.1:38721)
0926 09:28:58.966429 (+ 14016us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:58.966435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:58.966450 (+ 15us) server_negotiation.cc:407] Connection header received
0926 09:28:58.966488 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:58.966490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:58.966508 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:58.966609 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:28:58.966776 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.968551 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.970832 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.971735 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.972564 (+ 829us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:58.972660 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:58.972692 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 09:28:58.972742 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13974,"thread_start_us":43,"threads_started":1}
I20260926 09:28:58.974201 11721 tablet_service.cc:1467] Processing CreateTablet for tablet 7a9714552cea4764a76baac2147078b0 (DEFAULT_TABLE table=test-table [id=b7f2e6d2d3d8454eb55daa1d774ed3f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:28:58.974627 11721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a9714552cea4764a76baac2147078b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:28:58.979295 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:58.952870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33312 (local address 127.0.98.3:41169)
0926 09:28:58.961999 (+ 9129us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:58.962006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:58.962017 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:28:58.962058 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:58.962060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:58.962077 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:58.962173 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:28:58.962407 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.963538 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.965381 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:58.966259 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:58.966340 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:58.966422 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:58.979129 (+ 12707us) server_negotiation.cc:299] Negotiation successful
0926 09:28:58.979197 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9090,"thread_start_us":90,"threads_started":1}
I20260926 09:28:58.980224 12008 tablet_service.cc:1467] Processing CreateTablet for tablet 7a9714552cea4764a76baac2147078b0 (DEFAULT_TABLE table=test-table [id=b7f2e6d2d3d8454eb55daa1d774ed3f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:28:58.980588 12008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a9714552cea4764a76baac2147078b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:28:58.986723 12094 tablet_bootstrap.cc:492] T 7a9714552cea4764a76baac2147078b0 P 5c7b691b2150464fad0c9bc0a2b4c232: Bootstrap starting.
I20260926 09:28:58.987635 12094 tablet_bootstrap.cc:654] T 7a9714552cea4764a76baac2147078b0 P 5c7b691b2150464fad0c9bc0a2b4c232: Neither blocks nor log segments found. Creating new log.
I20260926 09:28:58.987888 12095 tablet_bootstrap.cc:492] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7: Bootstrap starting.
I20260926 09:28:58.988988 12095 tablet_bootstrap.cc:654] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7: Neither blocks nor log segments found. Creating new log.
I20260926 09:28:58.989431 12094 log.cc:824] T 7a9714552cea4764a76baac2147078b0 P 5c7b691b2150464fad0c9bc0a2b4c232: Log is configured to *not* fsync() on all Append() calls
I20260926 09:28:58.990092 12095 log.cc:824] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7: Log is configured to *not* fsync() on all Append() calls
I20260926 09:28:58.990545 12093 tablet_bootstrap.cc:492] T 7a9714552cea4764a76baac2147078b0 P 70287eea7b9a4d0a9bb9cd40b6a99b4f: Bootstrap starting.
I20260926 09:28:58.991792 12093 tablet_bootstrap.cc:654] T 7a9714552cea4764a76baac2147078b0 P 70287eea7b9a4d0a9bb9cd40b6a99b4f: Neither blocks nor log segments found. Creating new log.
I20260926 09:28:58.992187 12093 log.cc:824] T 7a9714552cea4764a76baac2147078b0 P 70287eea7b9a4d0a9bb9cd40b6a99b4f: Log is configured to *not* fsync() on all Append() calls
I20260926 09:28:58.993160 12094 tablet_bootstrap.cc:492] T 7a9714552cea4764a76baac2147078b0 P 5c7b691b2150464fad0c9bc0a2b4c232: No bootstrap required, opened a new log
I20260926 09:28:58.993319 12094 ts_tablet_manager.cc:1397] T 7a9714552cea4764a76baac2147078b0 P 5c7b691b2150464fad0c9bc0a2b4c232: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260926 09:28:58.998294 12094 raft_consensus.cc:348] T 7a9714552cea4764a76baac2147078b0 P 5c7b691b2150464fad0c9bc0a2b4c232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7b691b2150464fad0c9bc0a2b4c232" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38721 } } peers { permanent_uuid: "70287eea7b9a4d0a9bb9cd40b6a99b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38973 } } peers { permanent_uuid: "47e468cad6544156bb934e944f5e20e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41169 } }
I20260926 09:28:58.999060 12094 raft_consensus.cc:374] T 7a9714552cea4764a76baac2147078b0 P 5c7b691b2150464fad0c9bc0a2b4c232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:28:58.999202 12094 raft_consensus.cc:729] T 7a9714552cea4764a76baac2147078b0 P 5c7b691b2150464fad0c9bc0a2b4c232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c7b691b2150464fad0c9bc0a2b4c232, State: Initialized, Role: FOLLOWER
I20260926 09:28:58.999481 12094 consensus_queue.cc:260] T 7a9714552cea4764a76baac2147078b0 P 5c7b691b2150464fad0c9bc0a2b4c232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7b691b2150464fad0c9bc0a2b4c232" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38721 } } peers { permanent_uuid: "70287eea7b9a4d0a9bb9cd40b6a99b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38973 } } peers { permanent_uuid: "47e468cad6544156bb934e944f5e20e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41169 } }
I20260926 09:28:58.999940 12094 ts_tablet_manager.cc:1428] T 7a9714552cea4764a76baac2147078b0 P 5c7b691b2150464fad0c9bc0a2b4c232: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260926 09:28:59.000499 11802 heartbeater.cc:503] Master 127.0.98.62:42241 was elected leader, sending a full tablet report...
I20260926 09:28:59.002214 12095 tablet_bootstrap.cc:492] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7: No bootstrap required, opened a new log
I20260926 09:28:59.002388 12095 ts_tablet_manager.cc:1397] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260926 09:28:59.005311 12095 raft_consensus.cc:348] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7b691b2150464fad0c9bc0a2b4c232" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38721 } } peers { permanent_uuid: "70287eea7b9a4d0a9bb9cd40b6a99b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38973 } } peers { permanent_uuid: "47e468cad6544156bb934e944f5e20e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41169 } }
I20260926 09:28:59.005671 12095 raft_consensus.cc:374] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:28:59.005803 12095 raft_consensus.cc:729] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e468cad6544156bb934e944f5e20e7, State: Initialized, Role: FOLLOWER
I20260926 09:28:59.006055 12095 consensus_queue.cc:260] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7b691b2150464fad0c9bc0a2b4c232" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38721 } } peers { permanent_uuid: "70287eea7b9a4d0a9bb9cd40b6a99b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38973 } } peers { permanent_uuid: "47e468cad6544156bb934e944f5e20e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41169 } }
I20260926 09:28:59.006459 12095 ts_tablet_manager.cc:1428] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 09:28:59.006642 12074 heartbeater.cc:503] Master 127.0.98.62:42241 was elected leader, sending a full tablet report...
I20260926 09:28:59.008280 12093 tablet_bootstrap.cc:492] T 7a9714552cea4764a76baac2147078b0 P 70287eea7b9a4d0a9bb9cd40b6a99b4f: No bootstrap required, opened a new log
I20260926 09:28:59.008423 12093 ts_tablet_manager.cc:1397] T 7a9714552cea4764a76baac2147078b0 P 70287eea7b9a4d0a9bb9cd40b6a99b4f: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20260926 09:28:59.011641 12093 raft_consensus.cc:348] T 7a9714552cea4764a76baac2147078b0 P 70287eea7b9a4d0a9bb9cd40b6a99b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7b691b2150464fad0c9bc0a2b4c232" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38721 } } peers { permanent_uuid: "70287eea7b9a4d0a9bb9cd40b6a99b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38973 } } peers { permanent_uuid: "47e468cad6544156bb934e944f5e20e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41169 } }
I20260926 09:28:59.012075 12093 raft_consensus.cc:374] T 7a9714552cea4764a76baac2147078b0 P 70287eea7b9a4d0a9bb9cd40b6a99b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:28:59.012208 12093 raft_consensus.cc:729] T 7a9714552cea4764a76baac2147078b0 P 70287eea7b9a4d0a9bb9cd40b6a99b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70287eea7b9a4d0a9bb9cd40b6a99b4f, State: Initialized, Role: FOLLOWER
I20260926 09:28:59.012581 12093 consensus_queue.cc:260] T 7a9714552cea4764a76baac2147078b0 P 70287eea7b9a4d0a9bb9cd40b6a99b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7b691b2150464fad0c9bc0a2b4c232" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38721 } } peers { permanent_uuid: "70287eea7b9a4d0a9bb9cd40b6a99b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38973 } } peers { permanent_uuid: "47e468cad6544156bb934e944f5e20e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41169 } }
I20260926 09:28:59.013248 12093 ts_tablet_manager.cc:1428] T 7a9714552cea4764a76baac2147078b0 P 70287eea7b9a4d0a9bb9cd40b6a99b4f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 09:28:59.015295 11940 heartbeater.cc:503] Master 127.0.98.62:42241 was elected leader, sending a full tablet report...
Sep 26 09:28:59 dist-test-slave-dfzm krb5kdc[11114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:28:59.057350 12100 raft_consensus.cc:482] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:28:59.057492 12100 raft_consensus.cc:504] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7b691b2150464fad0c9bc0a2b4c232" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38721 } } peers { permanent_uuid: "70287eea7b9a4d0a9bb9cd40b6a99b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38973 } } peers { permanent_uuid: "47e468cad6544156bb934e944f5e20e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41169 } }
I20260926 09:28:59.057920 12100 leader_election.cc:288] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c7b691b2150464fad0c9bc0a2b4c232 (127.0.98.1:38721), 70287eea7b9a4d0a9bb9cd40b6a99b4f (127.0.98.2:38973)
I20260926 09:28:59.062951 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:59.058335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38721 (local address 127.0.98.3:36103)
0926 09:28:59.058391 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:59.058407 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:59.058490 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:59.059699 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:59.059700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:59.059702 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:28:59.059795 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:59.059798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:59.060663 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:59.060664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:59.061872 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:59.061875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:59.062678 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:59.062679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:59.062774 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:59.062779 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:28:59.062830 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 09:28:59.062875 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 09:28:59.063668 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:59.058664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36103 (local address 127.0.98.1:38721)
0926 09:28:59.059484 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:59.059486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:59.059496 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:28:59.059525 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:59.059527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:59.059539 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:59.059604 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:28:59.059881 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:59.060517 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:59.062013 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:59.062554 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:59.062629 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:59.062643 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:59.063525 (+ 882us) server_negotiation.cc:299] Negotiation successful
0926 09:28:59.063601 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793}
I20260926 09:28:59.064008 11756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9714552cea4764a76baac2147078b0" candidate_uuid: "47e468cad6544156bb934e944f5e20e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c7b691b2150464fad0c9bc0a2b4c232" is_pre_election: true
I20260926 09:28:59.064198 11756 raft_consensus.cc:2454] T 7a9714552cea4764a76baac2147078b0 P 5c7b691b2150464fad0c9bc0a2b4c232 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e468cad6544156bb934e944f5e20e7 in term 0.
I20260926 09:28:59.064465 11964 leader_election.cc:302] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e468cad6544156bb934e944f5e20e7, 5c7b691b2150464fad0c9bc0a2b4c232; no voters:
I20260926 09:28:59.064622 12100 raft_consensus.cc:2781] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:28:59.064683 12100 raft_consensus.cc:482] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:28:59.064728 12100 raft_consensus.cc:3037] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:28:59.065737 12100 raft_consensus.cc:504] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7b691b2150464fad0c9bc0a2b4c232" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38721 } } peers { permanent_uuid: "70287eea7b9a4d0a9bb9cd40b6a99b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38973 } } peers { permanent_uuid: "47e468cad6544156bb934e944f5e20e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41169 } }
I20260926 09:28:59.065877 12100 leader_election.cc:288] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7 [CANDIDATE]: Term 1 election: Requested vote from peers 5c7b691b2150464fad0c9bc0a2b4c232 (127.0.98.1:38721), 70287eea7b9a4d0a9bb9cd40b6a99b4f (127.0.98.2:38973)
I20260926 09:28:59.066118 11756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9714552cea4764a76baac2147078b0" candidate_uuid: "47e468cad6544156bb934e944f5e20e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c7b691b2150464fad0c9bc0a2b4c232"
I20260926 09:28:59.066195 11756 raft_consensus.cc:3037] T 7a9714552cea4764a76baac2147078b0 P 5c7b691b2150464fad0c9bc0a2b4c232 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:28:59.067129 11756 raft_consensus.cc:2454] T 7a9714552cea4764a76baac2147078b0 P 5c7b691b2150464fad0c9bc0a2b4c232 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e468cad6544156bb934e944f5e20e7 in term 1.
I20260926 09:28:59.067314 11964 leader_election.cc:302] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e468cad6544156bb934e944f5e20e7, 5c7b691b2150464fad0c9bc0a2b4c232; no voters:
I20260926 09:28:59.067402 12100 raft_consensus.cc:2781] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:28:59.067552 12100 raft_consensus.cc:686] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7 [term 1 LEADER]: Becoming Leader. State: Replica: 47e468cad6544156bb934e944f5e20e7, State: Running, Role: LEADER
I20260926 09:28:59.067655 12100 consensus_queue.cc:237] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7b691b2150464fad0c9bc0a2b4c232" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38721 } } peers { permanent_uuid: "70287eea7b9a4d0a9bb9cd40b6a99b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38973 } } peers { permanent_uuid: "47e468cad6544156bb934e944f5e20e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41169 } }
W20260926 09:28:59.067860 11803 tablet.cc:2369] T 7a9714552cea4764a76baac2147078b0 P 5c7b691b2150464fad0c9bc0a2b4c232: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:28:59.068550 11589 catalog_manager.cc:5495] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47e468cad6544156bb934e944f5e20e7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "47e468cad6544156bb934e944f5e20e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7b691b2150464fad0c9bc0a2b4c232" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70287eea7b9a4d0a9bb9cd40b6a99b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e468cad6544156bb934e944f5e20e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41169 } health_report { overall_health: HEALTHY } } }
I20260926 09:28:59.078088 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:59.058439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38973 (local address 127.0.98.3:42943)
0926 09:28:59.063040 (+ 4601us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:59.063052 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:28:59.063093 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:28:59.069746 (+ 6653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:28:59.069747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:28:59.069749 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:28:59.069795 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:59.069797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:59.070871 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:59.070873 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:59.074634 (+ 3761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:28:59.074638 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:28:59.077849 (+ 3211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:28:59.077851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:28:59.077915 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:59.077920 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:28:59.077964 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 09:28:59.078011 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4590,"thread_start_us":100,"threads_started":1}
I20260926 09:28:59.078709 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:59.059594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42943 (local address 127.0.98.2:38973)
0926 09:28:59.059663 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:59.059665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:59.063173 (+ 3508us) server_negotiation.cc:407] Connection header received
0926 09:28:59.063205 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:59.063207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:59.063223 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:59.063273 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:28:59.069917 (+ 6644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:59.070677 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:59.074757 (+ 4080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:59.075310 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:59.075385 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:59.075397 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:59.078585 (+ 3188us) server_negotiation.cc:299] Negotiation successful
0926 09:28:59.078646 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 09:28:59.081962 11875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9714552cea4764a76baac2147078b0" candidate_uuid: "47e468cad6544156bb934e944f5e20e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70287eea7b9a4d0a9bb9cd40b6a99b4f" is_pre_election: true
I20260926 09:28:59.082152 11875 raft_consensus.cc:2454] T 7a9714552cea4764a76baac2147078b0 P 70287eea7b9a4d0a9bb9cd40b6a99b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e468cad6544156bb934e944f5e20e7 in term 0.
I20260926 09:28:59.082298 11876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9714552cea4764a76baac2147078b0" candidate_uuid: "47e468cad6544156bb934e944f5e20e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70287eea7b9a4d0a9bb9cd40b6a99b4f"
I20260926 09:28:59.082369 11876 raft_consensus.cc:3037] T 7a9714552cea4764a76baac2147078b0 P 70287eea7b9a4d0a9bb9cd40b6a99b4f [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:28:59.082597 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:59.078946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.0.98.3:41169)
0926 09:28:59.079023 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 09:28:59.079025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:28:59.079179 (+ 154us) server_negotiation.cc:407] Connection header received
0926 09:28:59.079215 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:28:59.079216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:28:59.079251 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:28:59.079332 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:28:59.079556 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:59.080163 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:59.081282 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:28:59.081540 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:28:59.081890 (+ 350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:28:59.081929 (+ 39us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:28:59.082053 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:28:59.082129 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:28:59.082466 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 09:28:59.082489 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 09:28:59.083348 11876 raft_consensus.cc:2454] T 7a9714552cea4764a76baac2147078b0 P 70287eea7b9a4d0a9bb9cd40b6a99b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e468cad6544156bb934e944f5e20e7 in term 1.
I20260926 09:28:59.089355 11876 raft_consensus.cc:1261] T 7a9714552cea4764a76baac2147078b0 P 70287eea7b9a4d0a9bb9cd40b6a99b4f [term 1 FOLLOWER]: Refusing update from remote peer 47e468cad6544156bb934e944f5e20e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:28:59.089743 12103 consensus_queue.cc:1035] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70287eea7b9a4d0a9bb9cd40b6a99b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 09:28:59.090065 11941 tablet.cc:2369] T 7a9714552cea4764a76baac2147078b0 P 70287eea7b9a4d0a9bb9cd40b6a99b4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:28:59.091910 11756 raft_consensus.cc:1261] T 7a9714552cea4764a76baac2147078b0 P 5c7b691b2150464fad0c9bc0a2b4c232 [term 1 FOLLOWER]: Refusing update from remote peer 47e468cad6544156bb934e944f5e20e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:28:59.092218 12103 consensus_queue.cc:1035] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c7b691b2150464fad0c9bc0a2b4c232" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:28:59.105849 12106 mvcc.cc:205] Tried to move back new op lower bound from 7333539590506876928 to 7333539590421688320. Current Snapshot: MvccSnapshot[applied={T|T < 7333539590506876928}]
I20260926 09:28:59.107677 12109 mvcc.cc:205] Tried to move back new op lower bound from 7333539590506876928 to 7333539590421688320. Current Snapshot: MvccSnapshot[applied={T|T < 7333539590506876928}]
I20260926 09:28:59.107803 12107 mvcc.cc:205] Tried to move back new op lower bound from 7333539590506876928 to 7333539590421688320. Current Snapshot: MvccSnapshot[applied={T|T < 7333539590506876928}]
I20260926 09:28:59.115895 11589 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:28:59.116009 11589 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:28:59.117054 11589 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d164169f5f6546e2a26bc6d66244c305: Sending DeleteTablet for 3 replicas of tablet 7a9714552cea4764a76baac2147078b0
I20260926 09:28:59.117547 11721 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a9714552cea4764a76baac2147078b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:28:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60444
I20260926 09:28:59.117827 11863 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a9714552cea4764a76baac2147078b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:28:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59546
I20260926 09:28:59.118036 12008 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a9714552cea4764a76baac2147078b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:28:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33312
I20260926 09:28:59.118691 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 11671
I20260926 09:28:59.129719 12116 tablet_replica.cc:330] T 7a9714552cea4764a76baac2147078b0 P 70287eea7b9a4d0a9bb9cd40b6a99b4f: stopping tablet replica
I20260926 09:28:59.129958 12117 tablet_replica.cc:330] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7: stopping tablet replica
I20260926 09:28:59.130180 12117 raft_consensus.cc:2229] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:28:59.130374 12116 raft_consensus.cc:2229] T 7a9714552cea4764a76baac2147078b0 P 70287eea7b9a4d0a9bb9cd40b6a99b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:28:59.130592 12117 raft_consensus.cc:2258] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:28:59.130771 12116 raft_consensus.cc:2258] T 7a9714552cea4764a76baac2147078b0 P 70287eea7b9a4d0a9bb9cd40b6a99b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:28:59.131597 12116 ts_tablet_manager.cc:1905] T 7a9714552cea4764a76baac2147078b0 P 70287eea7b9a4d0a9bb9cd40b6a99b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:28:59.131929 12117 ts_tablet_manager.cc:1905] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:28:59.133648 12117 ts_tablet_manager.cc:1918] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:28:59.135291 12116 ts_tablet_manager.cc:1918] T 7a9714552cea4764a76baac2147078b0 P 70287eea7b9a4d0a9bb9cd40b6a99b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:28:59.135423 12116 log.cc:1196] T 7a9714552cea4764a76baac2147078b0 P 70287eea7b9a4d0a9bb9cd40b6a99b4f: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-1/wal/wals/7a9714552cea4764a76baac2147078b0
I20260926 09:28:59.135530 12117 log.cc:1196] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/7a9714552cea4764a76baac2147078b0
I20260926 09:28:59.135895 12116 ts_tablet_manager.cc:1939] T 7a9714552cea4764a76baac2147078b0 P 70287eea7b9a4d0a9bb9cd40b6a99b4f: Deleting consensus metadata
I20260926 09:28:59.136263 12117 ts_tablet_manager.cc:1939] T 7a9714552cea4764a76baac2147078b0 P 47e468cad6544156bb934e944f5e20e7: Deleting consensus metadata
I20260926 09:28:59.136806 11577 catalog_manager.cc:4841] TS 70287eea7b9a4d0a9bb9cd40b6a99b4f (127.0.98.2:38973): tablet 7a9714552cea4764a76baac2147078b0 (table test-table [id=b7f2e6d2d3d8454eb55daa1d774ed3f2]) successfully deleted
I20260926 09:28:59.137356 11577 catalog_manager.cc:4841] TS 47e468cad6544156bb934e944f5e20e7 (127.0.98.3:41169): tablet 7a9714552cea4764a76baac2147078b0 (table test-table [id=b7f2e6d2d3d8454eb55daa1d774ed3f2]) successfully deleted
W20260926 09:28:59.141227 11577 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38721 (error 108)
I20260926 09:28:59.142155 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:28:59.142007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38721 (local address 127.0.0.1:60446)
0926 09:28:59.142063 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 09:28:59.142119 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 09:28:59.142319 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 11808
W20260926 09:28:59.142860 11577 catalog_manager.cc:4584] TS 5c7b691b2150464fad0c9bc0a2b4c232 (127.0.98.1:38721): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7a9714552cea4764a76baac2147078b0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38721: connect: Connection refused (error 111)
I20260926 09:28:59.148216 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 11945
I20260926 09:28:59.167264 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 11558
I20260926 09:28:59.175033 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 09:28:59.519613 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 09:28:59.519717 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 09:28:59.864169 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 09:28:59.864 UTC [11144] LOG: received smart shutdown request
2026-09-26 09:28:59.867 UTC [11144] LOG: background worker "logical replication launcher" (PID 11153) exited with exit code 1
2026-09-26 09:28:59.867 UTC [11148] LOG: shutting down
2026-09-26 09:29:00.049 UTC [11144] LOG: database system is shut down
2026-09-26T09:29:00Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45675 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:29:00 dist-test-slave-dfzm krb5kdc[12127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:29:00 dist-test-slave-dfzm krb5kdc[12127](info): set up 2 sockets
Sep 26 09:29:00 dist-test-slave-dfzm krb5kdc[12127](info): commencing operation
krb5kdc: starting...
W20260926 09:29:01.201481 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:29:01 dist-test-slave-dfzm krb5kdc[12127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:29:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:29:01Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 09:29:01.368405 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskNX8gJ6/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/postgres -l logfile start
2026-09-26 09:29:03.909 UTC [12157] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 09:29:03.909 UTC [12157] LOG: listening on IPv4 address "127.0.98.20", port 51713
2026-09-26 09:29:03.912 UTC [12157] LOG: listening on Unix socket "/tmp/.s.PGSQL.51713"
2026-09-26 09:29:03.934 UTC [12160] LOG: database system was shut down at 2026-09-26 09:29:02 UTC
2026-09-26 09:29:03.938 UTC [12157] LOG: database system is ready to accept connections
I20260926 09:29:04.893428 392 mini_postgres.cc:96] Postgres bound to 51713
2026-09-26 09:29:04.900 UTC [12168] FATAL: database "slave" does not exist
127.0.98.20:51713 - accepting connections
I20260926 09:29:04.900525 392 mini_ranger.cc:161] Starting Ranger...
I20260926 09:29:04.914930 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 09:29:05.400544 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 09:29:05.400637 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ranger-admin
2026-09-26 09:29:05,672 [I] DB FLAVOR :POSTGRES
2026-09-26 09:29:05,673 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 09:29:05,673 [I] Checking connection..
2026-09-26 09:29:05,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 09:29:05,910 [I] Checking connection passed.
2026-09-26 09:29:05,911 [I] --------- Verifying version history table ---------
2026-09-26 09:29:05,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 09:29:06,142 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 09:29:06,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 09:29:06,396 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 09:29:06,396 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 09:29:06,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51713/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 09:29:06.643 UTC [12232] WARNING: there is no transaction in progress
2026-09-26 09:29:06,649 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 09:29:06,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 09:29:06,877 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 09:29:06,878 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 09:29:06,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 09:29:07,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 09:29:07,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dfzm.c.gcp-upstream.internal','N') ;"
2026-09-26 09:29:07,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 09:29:07,843 [I] Table x_portal_user does not exist in database ranger
2026-09-26 09:29:07,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 09:29:08,072 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 09:29:08,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 09:29:08,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 09:29:08,551 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 09:29:08,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51713/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 09:29:09.493 UTC [12340] WARNING: there is no transaction in progress
2026-09-26 09:29:09.509 UTC [12340] WARNING: there is no transaction in progress
2026-09-26 09:29:09.526 UTC [12340] WARNING: there is no transaction in progress
2026-09-26 09:29:09.542 UTC [12340] WARNING: there is no transaction in progress
2026-09-26 09:29:09.559 UTC [12340] WARNING: there is no transaction in progress
2026-09-26 09:29:09.576 UTC [12340] WARNING: there is no transaction in progress
2026-09-26 09:29:09.632 UTC [12340] WARNING: there is no transaction in progress
2026-09-26 09:29:09.639 UTC [12340] WARNING: there is no transaction in progress
2026-09-26 09:29:09.654 UTC [12340] WARNING: there is no transaction in progress
2026-09-26 09:29:09.665 UTC [12340] WARNING: there is no transaction in progress
2026-09-26 09:29:09.676 UTC [12340] WARNING: there is no transaction in progress
2026-09-26 09:29:09.690 UTC [12340] WARNING: there is no transaction in progress
2026-09-26 09:29:09.701 UTC [12340] WARNING: there is no transaction in progress
2026-09-26 09:29:09.709 UTC [12340] WARNING: there is no transaction in progress
2026-09-26 09:29:09.717 UTC [12340] WARNING: there is no transaction in progress
2026-09-26 09:29:09.724 UTC [12340] WARNING: there is no transaction in progress
2026-09-26 09:29:10,399 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 09:29:10,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 09:29:10,625 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 09:29:10,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 09:29:10,879 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 09:29:10,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 09:29:11,142 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 09:29:11,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 09:29:11,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 09:29:11,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dfzm.c.gcp-upstream.internal';"
2026-09-26 09:29:11,926 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 09:29:11,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51713/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 09:29:12,162 [I] DB_PATCHES have already been applied
I20260926 09:29:12.167480 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskNX8gJ6/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 09:29:12.167551 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 09:29:12.170650 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:36009
Sep 26, 2026 9:29:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 9:29:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 9:29:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 9:29:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 9:29:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 9:29:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 9:29:13 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 09:29:13.310741 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.410s user 0.000s sys 0.009s
Sep 26, 2026 9:29:33 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 9:29:33 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 9:29:33 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 9:29:33 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 9:29:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 9:29:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 9:29:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 9:29:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 9:29:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 9:29:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 9:29:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 9:29:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 9:29:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 9:29:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 9:29:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 9:29:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 9:29:33 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 9:29:34 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 09:29:34.770368 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 09:29:34 dist-test-slave-dfzm krb5kdc[12127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414974, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 09:29:34.854480 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 09:29:34.863969 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 09:29:35.770964 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 09:29:35.771054 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ranger-kms
I20260926 09:29:35.771076 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:51713
2026-09-26 09:29:35,931 [I] DB FLAVOR :POSTGRES
2026-09-26 09:29:35,932 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 09:29:35,932 [I] Checking connection
2026-09-26 09:29:35,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51713/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 09:29:36,162 [I] connection success
2026-09-26 09:29:36,162 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 09:29:36,162 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 09:29:36,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51713/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 09:29:36,401 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 09:29:36,401 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 09:29:36,402 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 09:29:36,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51713/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 09:29:36,656 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 09:29:37.131307 392 mini_ranger_kms.cc:310] Created kms service
I20260926 09:29:37.327509 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 09:29:37.374405 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 09:29:37 dist-test-slave-dfzm krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414974, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:29:37.775641 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 09:29:37.775839 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskNX8gJ6/build/release/bin/postgresql.jar:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskNX8gJ6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 09:29:37.775862 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 09:29:37.779515 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12520
I20260926 09:29:37.779600 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:39351
Sep 26, 2026 9:29:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 9:29:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 9:29:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 9:29:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 9:29:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 9:29:39 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 09:29:40.046064 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.192s user 0.005s sys 0.138s
I20260926 09:29:40.046166 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 09:29:40.046218 392 mini_ranger_kms.cc:401] 127.0.98.20:39351/kms/v1/keys
Sep 26 09:29:40 dist-test-slave-dfzm krb5kdc[12127](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790414980, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 09:29:40 dist-test-slave-dfzm krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414980, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 09:29:40 dist-test-slave-dfzm krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414980, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 09:29:43 dist-test-slave-dfzm krb5kdc[12127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:29:43.148638 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36741
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36741
--ranger_config_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39351/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:29:43.244720 12571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:29:43.244958 12571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:29:43.244985 12571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:29:43.246261 12571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:29:43.246310 12571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:29:43.246325 12571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:29:43.246366 12571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:29:43.246397 12571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:29:43.248031 12571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:39351/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36741
--ranger_config_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:29:43.248296 12571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:29:43.248535 12571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:29:43.254004 12571 server_base.cc:1034] running on GCE node
W20260926 09:29:43.253871 12576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:43.253871 12579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:43.253872 12577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:43.254407 12571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:29:43.254760 12571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:29:43.255923 12571 hybrid_clock.cc:648] HybridClock initialized: now 1790414983255899 us; error 41 us; skew 500 ppm
Sep 26 09:29:43 dist-test-slave-dfzm krb5kdc[12127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:43.258841 12571 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:29:43.259323 12571 webserver.cc:460] Webserver started at http://127.0.98.62:43491/ using document root <none> and password file <none>
I20260926 09:29:43.259558 12571 fs_manager.cc:359] Metadata directory not provided
I20260926 09:29:43.259621 12571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:29:43.259774 12571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 09:29:43 dist-test-slave-dfzm krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:29:43.315438 12571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "f963fdd4a465452295c73563e4967760"
format_stamp: "Formatted at 2026-09-26 09:29:43 on dist-test-slave-dfzm"
server_key: "0b4b5e31ec548578cd2d6e1670fa680f"
server_key_iv: "2a3f963daec4e0452362ba1e7c99783d"
server_key_version: "kuduclusterkey@0"
I20260926 09:29:43.315908 12571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "f963fdd4a465452295c73563e4967760"
format_stamp: "Formatted at 2026-09-26 09:29:43 on dist-test-slave-dfzm"
server_key: "0b4b5e31ec548578cd2d6e1670fa680f"
server_key_iv: "2a3f963daec4e0452362ba1e7c99783d"
server_key_version: "kuduclusterkey@0"
I20260926 09:29:43.317791 12571 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 09:29:43.332238 12586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:43.333920 12571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 09:29:43.334065 12571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "f963fdd4a465452295c73563e4967760"
format_stamp: "Formatted at 2026-09-26 09:29:43 on dist-test-slave-dfzm"
server_key: "0b4b5e31ec548578cd2d6e1670fa680f"
server_key_iv: "2a3f963daec4e0452362ba1e7c99783d"
server_key_version: "kuduclusterkey@0"
I20260926 09:29:43.334178 12571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:29:43.345140 12571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:29:43.347421 12571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:29:43.347611 12571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:29:43.352536 12571 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36741
I20260926 09:29:43.352614 12638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36741 every 8 connection(s)
I20260926 09:29:43.352988 12571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:29:43.353760 12639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:29:43.355454 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 12571
I20260926 09:29:43.355567 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:29:43.355736 392 external_mini_cluster.cc:1428] Setting key 2161741bc67eaf52e707443c5ad04225
I20260926 09:29:43.356345 12639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760: Bootstrap starting.
I20260926 09:29:43.357017 12639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760: Neither blocks nor log segments found. Creating new log.
I20260926 09:29:43.357389 12639 log.cc:824] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760: Log is configured to *not* fsync() on all Append() calls
I20260926 09:29:43.358237 12639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760: No bootstrap required, opened a new log
I20260926 09:29:43.360436 12639 raft_consensus.cc:348] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f963fdd4a465452295c73563e4967760" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36741 } }
I20260926 09:29:43.360670 12639 raft_consensus.cc:374] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:29:43.360726 12639 raft_consensus.cc:729] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f963fdd4a465452295c73563e4967760, State: Initialized, Role: FOLLOWER
I20260926 09:29:43.360883 12639 consensus_queue.cc:260] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f963fdd4a465452295c73563e4967760" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36741 } }
I20260926 09:29:43.360999 12639 raft_consensus.cc:388] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:29:43.361054 12639 raft_consensus.cc:482] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:29:43.361125 12639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:29:43.361989 12639 raft_consensus.cc:504] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f963fdd4a465452295c73563e4967760" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36741 } }
I20260926 09:29:43.362120 12639 leader_election.cc:302] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f963fdd4a465452295c73563e4967760; no voters:
I20260926 09:29:43.362317 12639 leader_election.cc:288] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:29:43.362371 12644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:29:43.362660 12644 raft_consensus.cc:686] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [term 1 LEADER]: Becoming Leader. State: Replica: f963fdd4a465452295c73563e4967760, State: Running, Role: LEADER
I20260926 09:29:43.362751 12639 sys_catalog.cc:564] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:29:43.362792 12644 consensus_queue.cc:237] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f963fdd4a465452295c73563e4967760" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36741 } }
I20260926 09:29:43.363139 12644 sys_catalog.cc:455] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f963fdd4a465452295c73563e4967760. Latest consensus state: current_term: 1 leader_uuid: "f963fdd4a465452295c73563e4967760" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f963fdd4a465452295c73563e4967760" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36741 } } }
I20260926 09:29:43.363207 12644 sys_catalog.cc:458] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [sys.catalog]: This master's current role is: LEADER
I20260926 09:29:43.363420 12648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 09:29:43 dist-test-slave-dfzm krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:29:43.363973 12648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:29:43.364369 12639 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 09:29:43.364149 12646 sys_catalog.cc:455] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f963fdd4a465452295c73563e4967760" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f963fdd4a465452295c73563e4967760" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36741 } } }
I20260926 09:29:43.367400 12646 sys_catalog.cc:458] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [sys.catalog]: This master's current role is: LEADER
I20260926 09:29:43.370155 12648 catalog_manager.cc:1347] Generated new cluster ID: 554231aecb6f4603a662ca109b7547e1
I20260926 09:29:43.370204 12648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:29:43.370456 12642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:43.356285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53920 (local address 127.0.98.62:36741)
0926 09:29:43.356464 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:43.356474 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:43.356505 (+ 31us) server_negotiation.cc:407] Connection header received
0926 09:29:43.357067 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:43.357077 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:43.357143 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:43.357306 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:43.358124 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:43.359532 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:43.360650 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:43.361281 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:43.361395 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:43.364550 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:43.364554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:43.364555 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:43.364568 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:43.366923 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:43.368604 (+ 1681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:43.368608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:43.368608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:43.368715 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:43.368942 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:43.368942 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:43.368943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:43.369234 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:43.369379 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:43.369869 (+ 490us) server_negotiation.cc:299] Negotiation successful
0926 09:29:43.369987 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260926 09:29:43.390151 12648 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:29:43.391044 12648 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:29:43.398615 12648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760: Generated new TSK 0
I20260926 09:29:43.398931 12648 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:29:44.149286 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39351/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36741
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:29:44.270844 12684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:29:44.271215 12684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:29:44.271286 12684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:29:44.273037 12684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:29:44.273109 12684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:29:44.273149 12684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:29:44.275293 12684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:39351/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:29:44.275578 12684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:29:44.275861 12684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:29:44.282244 12692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:44.282258 12689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:44.282630 12690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:44.283668 12684 server_base.cc:1034] running on GCE node
I20260926 09:29:44.283902 12684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:29:44.284191 12684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:29:44.285367 12684 hybrid_clock.cc:648] HybridClock initialized: now 1790414984285339 us; error 40 us; skew 500 ppm
Sep 26 09:29:44 dist-test-slave-dfzm krb5kdc[12127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:44.288657 12684 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:29:44.289256 12684 webserver.cc:460] Webserver started at http://127.0.98.1:36665/ using document root <none> and password file <none>
I20260926 09:29:44.289611 12684 fs_manager.cc:359] Metadata directory not provided
I20260926 09:29:44.289727 12684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:29:44.289896 12684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 09:29:44 dist-test-slave-dfzm krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:29:44.307015 12684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "02be29d50ad8404381782363b3f4817a"
format_stamp: "Formatted at 2026-09-26 09:29:44 on dist-test-slave-dfzm"
server_key: "375810f199c2519205dc5439a538ecd3"
server_key_iv: "9e6a50663fda8bbc46725467cb0b9409"
server_key_version: "kuduclusterkey@0"
I20260926 09:29:44.307482 12684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "02be29d50ad8404381782363b3f4817a"
format_stamp: "Formatted at 2026-09-26 09:29:44 on dist-test-slave-dfzm"
server_key: "375810f199c2519205dc5439a538ecd3"
server_key_iv: "9e6a50663fda8bbc46725467cb0b9409"
server_key_version: "kuduclusterkey@0"
I20260926 09:29:44.309424 12684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260926 09:29:44.322032 12699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:44.322276 12684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:29:44.322371 12684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "02be29d50ad8404381782363b3f4817a"
format_stamp: "Formatted at 2026-09-26 09:29:44 on dist-test-slave-dfzm"
server_key: "375810f199c2519205dc5439a538ecd3"
server_key_iv: "9e6a50663fda8bbc46725467cb0b9409"
server_key_version: "kuduclusterkey@0"
I20260926 09:29:44.322464 12684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:29:44.371099 12684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:29:44.373863 12684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:29:44.374014 12684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:29:44.374274 12684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:29:44.374681 12684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:29:44.374727 12684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:44.374758 12684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:29:44.374773 12684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:44.385519 12684 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34139
I20260926 09:29:44.385818 12814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34139 every 8 connection(s)
I20260926 09:29:44.386152 12684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:29:44.392536 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 12684
I20260926 09:29:44.392647 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:29:44.392819 392 external_mini_cluster.cc:1428] Setting key 1d723adbb3e87bb82ff67e138f12c6f9
Sep 26 09:29:44 dist-test-slave-dfzm krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:29:44.405961 12817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:44.387178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36741 (local address 127.0.98.1:36553)
0926 09:29:44.394238 (+ 7060us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:44.394311 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:44.395095 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:44.395646 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:44.395651 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:44.395912 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:29:44.396210 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:44.396216 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:44.397216 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:44.397217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:44.398480 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:44.398483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:44.399055 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:44.399056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:44.399200 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:44.399879 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:29:44.399890 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:29:44.401577 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:29:44.404020 (+ 2443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:44.404029 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:44.404031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:44.404382 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:44.404660 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:44.404661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:44.404661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:44.404812 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:44.405152 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:29:44.405159 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:29:44.405349 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 09:29:44.405489 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 09:29:44.405699 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6996,"thread_start_us":110,"threads_started":1}
I20260926 09:29:44.406818 12816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:44.387203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36553 (local address 127.0.98.62:36741)
0926 09:29:44.395386 (+ 8183us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:44.395389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:44.395397 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:29:44.395421 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:44.395421 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:44.395434 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:44.395496 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:44.396323 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:44.397104 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:44.398581 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:44.398890 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:44.398955 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:44.401695 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:44.401697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:44.401698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:44.401707 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:44.403900 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:44.404502 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:44.404504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:44.404504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:44.404577 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:44.404898 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:44.404899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:44.404899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:44.405050 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:44.405121 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:44.406607 (+ 1486us) server_negotiation.cc:299] Negotiation successful
0926 09:29:44.406707 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8151,"thread_start_us":83,"threads_started":1}
I20260926 09:29:44.407588 12815 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36741
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 09:29:44.407697 12815 heartbeater.cc:464] Registering TS with master...
I20260926 09:29:44.407903 12815 heartbeater.cc:511] Master 127.0.98.62:36741 requested a full tablet report, sending...
I20260926 09:29:44.408514 12603 ts_manager.cc:194] Registered new tserver with Master: 02be29d50ad8404381782363b3f4817a (127.0.98.1:34139)
I20260926 09:29:44.409425 12603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:29:44.429209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39351/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36741
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:29:44 dist-test-slave-dfzm krb5kdc[12127](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790414984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 09:29:44.569276 12821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:29:44.569667 12821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:29:44.569748 12821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:29:44.571621 12821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:29:44.571688 12821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:29:44.571730 12821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:29:44.574071 12821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:39351/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:29:44.574545 12821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:29:44.574914 12821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:29:44.584589 12831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:44.584787 12829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:44.585178 12828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:44.585371 12821 server_base.cc:1034] running on GCE node
I20260926 09:29:44.585803 12821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:29:44.586112 12821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:29:44.587291 12821 hybrid_clock.cc:648] HybridClock initialized: now 1790414984587265 us; error 35 us; skew 500 ppm
Sep 26 09:29:44 dist-test-slave-dfzm krb5kdc[12127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:44.590801 12821 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:29:44.591398 12821 webserver.cc:460] Webserver started at http://127.0.98.2:36883/ using document root <none> and password file <none>
I20260926 09:29:44.591704 12821 fs_manager.cc:359] Metadata directory not provided
I20260926 09:29:44.591807 12821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:29:44.591964 12821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 09:29:44 dist-test-slave-dfzm krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:29:44.606865 12821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "29b964e356c24c7ca5022796c8abc01b"
format_stamp: "Formatted at 2026-09-26 09:29:44 on dist-test-slave-dfzm"
server_key: "bd2ccca1a4d32d16a23359c219d1aad7"
server_key_iv: "e4ec606b56d76a505d32cf9f2d8d8d69"
server_key_version: "kuduclusterkey@0"
I20260926 09:29:44.607322 12821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "29b964e356c24c7ca5022796c8abc01b"
format_stamp: "Formatted at 2026-09-26 09:29:44 on dist-test-slave-dfzm"
server_key: "bd2ccca1a4d32d16a23359c219d1aad7"
server_key_iv: "e4ec606b56d76a505d32cf9f2d8d8d69"
server_key_version: "kuduclusterkey@0"
I20260926 09:29:44.609170 12821 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260926 09:29:44.629405 12838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:44.635941 12821 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.001s sys 0.000s
I20260926 09:29:44.636087 12821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "29b964e356c24c7ca5022796c8abc01b"
format_stamp: "Formatted at 2026-09-26 09:29:44 on dist-test-slave-dfzm"
server_key: "bd2ccca1a4d32d16a23359c219d1aad7"
server_key_iv: "e4ec606b56d76a505d32cf9f2d8d8d69"
server_key_version: "kuduclusterkey@0"
I20260926 09:29:44.636209 12821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:29:44.665180 12821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:29:44.667662 12821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:29:44.667821 12821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:29:44.668054 12821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:29:44.668452 12821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:29:44.668503 12821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:44.668560 12821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:29:44.668586 12821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:44.675941 12821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42095
I20260926 09:29:44.676020 12952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42095 every 8 connection(s)
I20260926 09:29:44.676474 12821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:29:44 dist-test-slave-dfzm krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:29:44.685062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 12821
I20260926 09:29:44.685173 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:29:44.685375 392 external_mini_cluster.cc:1428] Setting key 9706e68b8ef9073c881973e833fb80fd
I20260926 09:29:44.690263 12816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:44.677641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34663 (local address 127.0.98.62:36741)
0926 09:29:44.677695 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:44.677698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:44.677710 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:29:44.678225 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:44.678227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:44.678242 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:44.678334 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:44.678979 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:44.679768 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:44.681366 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:44.681717 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:44.681803 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:44.684573 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:44.684575 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:44.684575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:44.684584 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:44.686437 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:44.689057 (+ 2620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:44.689058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:44.689058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:44.689120 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:44.689404 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:44.689405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:44.689405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:44.689560 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:44.689645 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:44.690066 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 09:29:44.690164 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:29:44.690815 12955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:44.677292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36741 (local address 127.0.98.2:34663)
0926 09:29:44.677505 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:44.677566 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:44.678104 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:44.678436 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:44.678440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:44.678657 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:29:44.678870 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:44.678874 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:44.679886 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:44.679887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:44.681258 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:44.681261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:44.681837 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:44.681838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:44.681988 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:44.682685 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:29:44.682696 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:29:44.684445 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:29:44.688608 (+ 4163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:44.688617 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:44.688619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:44.688954 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:44.689192 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:44.689193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:44.689193 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:44.689328 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:44.689696 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:29:44.689703 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:29:44.689896 (+ 193us) client_negotiation.cc:769] Sending connection context
0926 09:29:44.690375 (+ 479us) client_negotiation.cc:240] Negotiation successful
0926 09:29:44.690562 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":100,"threads_started":1}
I20260926 09:29:44.691284 12953 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36741
I20260926 09:29:44.691395 12953 heartbeater.cc:464] Registering TS with master...
I20260926 09:29:44.691598 12953 heartbeater.cc:511] Master 127.0.98.62:36741 requested a full tablet report, sending...
I20260926 09:29:44.692020 12603 ts_manager.cc:194] Registered new tserver with Master: 29b964e356c24c7ca5022796c8abc01b (127.0.98.2:42095)
I20260926 09:29:44.692543 12603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34663
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:29:44.716347 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39351/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36741
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:29:44.827597 12958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:29:44.827837 12958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:29:44.827878 12958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:29:44.829061 12958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:29:44.829138 12958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:29:44.829185 12958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:29:44.830727 12958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:39351/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:29:44.831002 12958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:29:44.831238 12958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:29:44.839883 12964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:44.841677 12963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:44.841939 12966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:44.843387 12958 server_base.cc:1034] running on GCE node
I20260926 09:29:44.843698 12958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:29:44.844004 12958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:29:44.845199 12958 hybrid_clock.cc:648] HybridClock initialized: now 1790414984845172 us; error 46 us; skew 500 ppm
Sep 26 09:29:44 dist-test-slave-dfzm krb5kdc[12127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:44.847965 12958 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:29:44.848485 12958 webserver.cc:460] Webserver started at http://127.0.98.3:38495/ using document root <none> and password file <none>
I20260926 09:29:44.848752 12958 fs_manager.cc:359] Metadata directory not provided
I20260926 09:29:44.848822 12958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:29:44.848963 12958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 09:29:44 dist-test-slave-dfzm krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:29:44.869439 12958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "75a0582140694938800476f012de866f"
format_stamp: "Formatted at 2026-09-26 09:29:44 on dist-test-slave-dfzm"
server_key: "8a7a129682e524ba540f246d84811cfb"
server_key_iv: "e2ff70dd5cf0a3a5906c11741da7a06e"
server_key_version: "kuduclusterkey@0"
I20260926 09:29:44.869918 12958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "75a0582140694938800476f012de866f"
format_stamp: "Formatted at 2026-09-26 09:29:44 on dist-test-slave-dfzm"
server_key: "8a7a129682e524ba540f246d84811cfb"
server_key_iv: "e2ff70dd5cf0a3a5906c11741da7a06e"
server_key_version: "kuduclusterkey@0"
I20260926 09:29:44.872433 12958 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 09:29:44.886215 12973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:44.886409 12958 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:29:44.886504 12958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "75a0582140694938800476f012de866f"
format_stamp: "Formatted at 2026-09-26 09:29:44 on dist-test-slave-dfzm"
server_key: "8a7a129682e524ba540f246d84811cfb"
server_key_iv: "e2ff70dd5cf0a3a5906c11741da7a06e"
server_key_version: "kuduclusterkey@0"
I20260926 09:29:44.886616 12958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:29:44.909416 12958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:29:44.912297 12958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:29:44.912464 12958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:29:44.912694 12958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:29:44.913151 12958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:29:44.913208 12958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:44.913249 12958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:29:44.913275 12958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:44.928453 12958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36397
I20260926 09:29:44.928938 12958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:29:44.933630 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 12958
I20260926 09:29:44.933760 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:29:44.933967 392 external_mini_cluster.cc:1428] Setting key a05038bca8cf0e907e250e47aeab36d1
I20260926 09:29:44.935113 13086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36397 every 8 connection(s)
Sep 26 09:29:44 dist-test-slave-dfzm krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:29:44.946794 12816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:44.934147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32833 (local address 127.0.98.62:36741)
0926 09:29:44.934368 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:44.934371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:44.935598 (+ 1227us) server_negotiation.cc:407] Connection header received
0926 09:29:44.936343 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:44.936345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:44.936356 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:44.936386 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:44.937293 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:44.938138 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:44.939669 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:44.939964 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:44.940012 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:44.942735 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:44.942737 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:44.942737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:44.942744 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:44.944481 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:44.945013 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:44.945015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:44.945015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:44.945091 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:44.946024 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:44.946025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:44.946025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:44.946173 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:44.946216 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:44.946616 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 09:29:44.946700 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260926 09:29:44.947607 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:44.933136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36741 (local address 127.0.98.3:32833)
0926 09:29:44.935474 (+ 2338us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:44.935517 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:44.936191 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:44.936579 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:44.936583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:44.936837 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:29:44.937184 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:44.937192 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:44.938279 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:44.938280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:44.939563 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:44.939566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:44.940136 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:44.940137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:44.940275 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:44.940910 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:29:44.940920 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:29:44.942627 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:29:44.944581 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:44.944588 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:44.944590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:44.944901 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:44.945792 (+ 891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:44.945793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:44.945794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:44.945940 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:44.946263 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:29:44.946269 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:29:44.946451 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 09:29:44.947187 (+ 736us) client_negotiation.cc:240] Negotiation successful
0926 09:29:44.947374 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2296,"thread_start_us":91,"threads_started":1}
I20260926 09:29:44.948309 13087 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36741
I20260926 09:29:44.948405 13087 heartbeater.cc:464] Registering TS with master...
I20260926 09:29:44.948596 13087 heartbeater.cc:511] Master 127.0.98.62:36741 requested a full tablet report, sending...
I20260926 09:29:44.948987 12603 ts_manager.cc:194] Registered new tserver with Master: 75a0582140694938800476f012de866f (127.0.98.3:36397)
I20260926 09:29:44.949474 12603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32833
I20260926 09:29:44.956491 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:29:44.957155 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 12684
I20260926 09:29:44.965010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 12821
I20260926 09:29:44.971961 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 12958
I20260926 09:29:44.979101 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 12571
I20260926 09:29:44.984575 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36741
--webserver_interface=127.0.98.62
--webserver_port=43491
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36741
--ranger_config_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39351/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39351/kms with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:29:45.066859 13090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:29:45.067225 13090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:29:45.067272 13090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:29:45.068537 13090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:29:45.068591 13090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:29:45.068607 13090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:29:45.068619 13090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:29:45.068632 13090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:29:45.070642 13090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39351/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36741
--ranger_config_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:29:45.070924 13090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:29:45.071178 13090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:29:45.076174 13095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:45.076177 13096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:45.076267 13098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:45.076551 13090 server_base.cc:1034] running on GCE node
I20260926 09:29:45.076813 13090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:29:45.077126 13090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:29:45.078320 13090 hybrid_clock.cc:648] HybridClock initialized: now 1790414985078297 us; error 37 us; skew 500 ppm
Sep 26 09:29:45 dist-test-slave-dfzm krb5kdc[12127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:45.081547 13090 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:29:45.082109 13090 webserver.cc:460] Webserver started at http://127.0.98.62:43491/ using document root <none> and password file <none>
I20260926 09:29:45.082348 13090 fs_manager.cc:359] Metadata directory not provided
I20260926 09:29:45.082427 13090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 09:29:45 dist-test-slave-dfzm krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:29:45.139780 13090 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:29:45.140574 13106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:45.140763 13090 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:29:45.140854 13090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "f963fdd4a465452295c73563e4967760"
format_stamp: "Formatted at 2026-09-26 09:29:43 on dist-test-slave-dfzm"
server_key: "0b4b5e31ec548578cd2d6e1670fa680f"
server_key_iv: "2a3f963daec4e0452362ba1e7c99783d"
server_key_version: "kuduclusterkey@0"
I20260926 09:29:45.141220 13090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:29:45.162986 13090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:29:45.164925 13090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:29:45.165084 13090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:29:45.170060 13090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36741
I20260926 09:29:45.170082 13158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36741 every 8 connection(s)
I20260926 09:29:45.170604 13090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:29:45.172163 13159 sys_catalog.cc:263] Verifying existing consensus state
I20260926 09:29:45.173002 13159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760: Bootstrap starting.
I20260926 09:29:45.180496 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 13090
I20260926 09:29:45.180826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39351/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34139
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36741
--webserver_port=36665
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39351/kms with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:29:45.182914 13159 log.cc:824] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760: Log is configured to *not* fsync() on all Append() calls
I20260926 09:29:45.185127 13159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:29:45.185570 13159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760: Bootstrap complete.
I20260926 09:29:45.189028 13159 raft_consensus.cc:348] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f963fdd4a465452295c73563e4967760" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36741 } }
I20260926 09:29:45.189368 13159 raft_consensus.cc:729] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f963fdd4a465452295c73563e4967760, State: Initialized, Role: FOLLOWER
I20260926 09:29:45.189568 13159 consensus_queue.cc:260] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f963fdd4a465452295c73563e4967760" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36741 } }
I20260926 09:29:45.189661 13159 raft_consensus.cc:388] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:29:45.189738 13159 raft_consensus.cc:482] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:29:45.189813 13159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [term 1 FOLLOWER]: Advancing to term 2
I20260926 09:29:45.191183 13159 raft_consensus.cc:504] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f963fdd4a465452295c73563e4967760" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36741 } }
I20260926 09:29:45.191296 13159 leader_election.cc:302] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f963fdd4a465452295c73563e4967760; no voters:
I20260926 09:29:45.191499 13159 leader_election.cc:288] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 09:29:45.191569 13163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 09:29:45.191792 13163 raft_consensus.cc:686] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [term 2 LEADER]: Becoming Leader. State: Replica: f963fdd4a465452295c73563e4967760, State: Running, Role: LEADER
I20260926 09:29:45.191828 13159 sys_catalog.cc:564] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:29:45.192227 13163 consensus_queue.cc:237] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f963fdd4a465452295c73563e4967760" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36741 } }
I20260926 09:29:45.192488 13163 sys_catalog.cc:455] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f963fdd4a465452295c73563e4967760" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f963fdd4a465452295c73563e4967760" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36741 } } }
I20260926 09:29:45.192538 13163 sys_catalog.cc:458] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [sys.catalog]: This master's current role is: LEADER
I20260926 09:29:45.192632 13163 sys_catalog.cc:455] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f963fdd4a465452295c73563e4967760. Latest consensus state: current_term: 2 leader_uuid: "f963fdd4a465452295c73563e4967760" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f963fdd4a465452295c73563e4967760" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36741 } } }
I20260926 09:29:45.192665 13163 sys_catalog.cc:458] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760 [sys.catalog]: This master's current role is: LEADER
I20260926 09:29:45.193301 13171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:29:45.193531 13159 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 09:29:45.193917 13171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:29:45.194041 13171 catalog_manager.cc:1259] Loaded cluster ID: 554231aecb6f4603a662ca109b7547e1
I20260926 09:29:45.194094 13171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:29:45.195320 13171 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:29:45.195503 13171 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760: Loaded TSK: 0
I20260926 09:29:45.195714 13171 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 09:29:45.299999 13161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:29:45.300412 13161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:29:45.300518 13161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:29:45.302573 13161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:29:45.302820 13161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:29:45.302942 13161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:29:45.305313 13161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39351/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36665
--tserver_master_addrs=127.0.98.62:36741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:29:45.305827 13161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:29:45.306142 13161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:29:45.312752 13197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:45.313472 13161 server_base.cc:1034] running on GCE node
W20260926 09:29:45.320521 13196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:45.321280 13199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:45.321626 13161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:29:45.321982 13161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:29:45.325255 13161 hybrid_clock.cc:648] HybridClock initialized: now 1790414985325218 us; error 45 us; skew 500 ppm
Sep 26 09:29:45 dist-test-slave-dfzm krb5kdc[12127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:45.328719 13161 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:29:45.329276 13161 webserver.cc:460] Webserver started at http://127.0.98.1:36665/ using document root <none> and password file <none>
I20260926 09:29:45.329530 13161 fs_manager.cc:359] Metadata directory not provided
I20260926 09:29:45.329627 13161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 09:29:45 dist-test-slave-dfzm krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:29:45.362131 13161 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.001s sys 0.000s
I20260926 09:29:45.365770 13207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:45.366034 13161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260926 09:29:45.366128 13161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "02be29d50ad8404381782363b3f4817a"
format_stamp: "Formatted at 2026-09-26 09:29:44 on dist-test-slave-dfzm"
server_key: "375810f199c2519205dc5439a538ecd3"
server_key_iv: "9e6a50663fda8bbc46725467cb0b9409"
server_key_version: "kuduclusterkey@0"
I20260926 09:29:45.366597 13161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:29:45.391165 13161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:29:45.393988 13161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:29:45.394203 13161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:29:45.394474 13161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:29:45.394878 13161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:29:45.394936 13161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:45.394980 13161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:29:45.395007 13161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:45.402886 13161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34139
I20260926 09:29:45.402982 13320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34139 every 8 connection(s)
I20260926 09:29:45.403407 13161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:29:45.404639 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 13161
I20260926 09:29:45.404970 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39351/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42095
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36741
--webserver_port=36883
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39351/kms with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:29:45 dist-test-slave-dfzm krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:29:45.417874 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:45.404478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36741 (local address 127.0.98.1:38057)
0926 09:29:45.405715 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:45.405766 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:45.406443 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:45.406960 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:45.406964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:45.407243 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:29:45.407538 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:45.407545 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:45.408922 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:45.408923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:45.410422 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:45.410425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:45.411404 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:45.411405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:45.411546 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:45.412192 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:29:45.412201 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:29:45.413695 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:29:45.415864 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:45.415871 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:45.415873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:45.416244 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:45.416549 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:45.416550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:45.416551 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:45.416678 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:45.417108 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:29:45.417124 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:29:45.417283 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 09:29:45.417401 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 09:29:45.417590 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1185,"thread_start_us":86,"threads_started":1}
I20260926 09:29:45.417862 13324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:45.404864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38057 (local address 127.0.98.62:36741)
0926 09:29:45.405204 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:45.405215 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:45.405911 (+ 696us) server_negotiation.cc:407] Connection header received
0926 09:29:45.406685 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:45.406692 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:45.406740 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:45.406869 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:45.407690 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:45.408804 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:45.410524 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:45.411282 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:45.411424 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:45.413819 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:45.413823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:45.413824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:45.413837 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:45.415746 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:45.416370 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:45.416379 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:45.416379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:45.416469 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:45.416762 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:45.416763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:45.416764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:45.417015 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:45.417114 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:45.417481 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 09:29:45.417648 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":102,"threads_started":1}
I20260926 09:29:45.419242 13321 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36741
I20260926 09:29:45.419379 13321 heartbeater.cc:464] Registering TS with master...
I20260926 09:29:45.419582 13321 heartbeater.cc:511] Master 127.0.98.62:36741 requested a full tablet report, sending...
W20260926 09:29:45.420481 13321 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36741 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 09:29:45.420557 13321 heartbeater.cc:464] Registering TS with master...
I20260926 09:29:45.420634 13321 heartbeater.cc:511] Master 127.0.98.62:36741 requested a full tablet report, sending...
I20260926 09:29:45.421054 13321 heartbeater.cc:464] Registering TS with master...
I20260926 09:29:45.421171 13321 heartbeater.cc:511] Master 127.0.98.62:36741 requested a full tablet report, sending...
W20260926 09:29:45.421463 13321 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 09:29:45.534148 13325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:29:45.534452 13325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:29:45.534498 13325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:29:45.536227 13325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:29:45.536301 13325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:29:45.536350 13325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:29:45.538604 13325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39351/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36883
--tserver_master_addrs=127.0.98.62:36741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:29:45.538877 13325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:29:45.539156 13325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:29:45.546517 13330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:45.546517 13331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:45.546542 13333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:45.547075 13325 server_base.cc:1034] running on GCE node
I20260926 09:29:45.547307 13325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:29:45.547616 13325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:29:45.550899 13325 hybrid_clock.cc:648] HybridClock initialized: now 1790414985550437 us; error 473 us; skew 500 ppm
Sep 26 09:29:45 dist-test-slave-dfzm krb5kdc[12127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:45.554286 13325 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:29:45.554809 13325 webserver.cc:460] Webserver started at http://127.0.98.2:36883/ using document root <none> and password file <none>
I20260926 09:29:45.555117 13325 fs_manager.cc:359] Metadata directory not provided
I20260926 09:29:45.555202 13325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 09:29:45 dist-test-slave-dfzm krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:29:45.576719 13325 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:29:45.577664 13341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:45.577888 13325 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:29:45.577993 13325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "29b964e356c24c7ca5022796c8abc01b"
format_stamp: "Formatted at 2026-09-26 09:29:44 on dist-test-slave-dfzm"
server_key: "bd2ccca1a4d32d16a23359c219d1aad7"
server_key_iv: "e4ec606b56d76a505d32cf9f2d8d8d69"
server_key_version: "kuduclusterkey@0"
I20260926 09:29:45.578497 13325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:29:45.591423 13325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:29:45.594079 13325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:29:45.594262 13325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:29:45.594547 13325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:29:45.594938 13325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:29:45.594995 13325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:45.595032 13325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:29:45.595064 13325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:45.603268 13325 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42095
I20260926 09:29:45.603413 13454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42095 every 8 connection(s)
I20260926 09:29:45.603767 13325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:29:45.605536 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 13325
I20260926 09:29:45.605868 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39351/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36397
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36741
--webserver_port=38495
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39351/kms with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:29:45 dist-test-slave-dfzm krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:29:45.619789 13324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:45.604500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35745 (local address 127.0.98.62:36741)
0926 09:29:45.604726 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:45.604729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:45.604954 (+ 225us) server_negotiation.cc:407] Connection header received
0926 09:29:45.607833 (+ 2879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:45.607834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:45.607853 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:45.607889 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:45.608747 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:45.609568 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:45.612552 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:45.612922 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:45.612973 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:45.615850 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:45.615851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:45.615852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:45.615861 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:45.618042 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:45.618610 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:45.618611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:45.618611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:45.618675 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:45.619004 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:45.619005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:45.619005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:45.619139 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:45.619206 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:45.619588 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 09:29:45.619676 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260926 09:29:45.620411 13457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:45.604500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36741 (local address 127.0.98.2:35745)
0926 09:29:45.604808 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:45.604849 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:45.607674 (+ 2825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:45.608095 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:45.608099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:45.608361 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:29:45.608638 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:45.608644 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:45.609704 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:45.609705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:45.612435 (+ 2730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:45.612439 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:45.613084 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:45.613085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:45.613238 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:45.613933 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:29:45.613944 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:29:45.615712 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:29:45.618140 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:45.618148 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:45.618150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:45.618501 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:45.618753 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:45.618754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:45.618755 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:45.618924 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:45.619221 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:29:45.619229 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:29:45.619417 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 09:29:45.619565 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 09:29:45.619751 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":78,"threads_started":1}
I20260926 09:29:45.621018 13455 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36741
I20260926 09:29:45.621114 13455 heartbeater.cc:464] Registering TS with master...
I20260926 09:29:45.621300 13455 heartbeater.cc:511] Master 127.0.98.62:36741 requested a full tablet report, sending...
W20260926 09:29:45.622069 13455 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36741 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 09:29:45.622136 13455 heartbeater.cc:464] Registering TS with master...
I20260926 09:29:45.622193 13455 heartbeater.cc:511] Master 127.0.98.62:36741 requested a full tablet report, sending...
I20260926 09:29:45.622635 13455 heartbeater.cc:464] Registering TS with master...
I20260926 09:29:45.622702 13455 heartbeater.cc:511] Master 127.0.98.62:36741 requested a full tablet report, sending...
W20260926 09:29:45.623091 13455 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 09:29:45.709251 13458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:29:45.709547 13458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:29:45.709613 13458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:29:45.711297 13458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:29:45.711378 13458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:29:45.711424 13458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:29:45.713007 13458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:39351/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38495
--tserver_master_addrs=127.0.98.62:36741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:29:45.713320 13458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:29:45.713559 13458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:29:45.719712 13466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:45.720479 13464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:45.720854 13458 server_base.cc:1034] running on GCE node
W20260926 09:29:45.721236 13463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:45.721508 13458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:29:45.721822 13458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:29:45.722972 13458 hybrid_clock.cc:648] HybridClock initialized: now 1790414985722955 us; error 37 us; skew 500 ppm
Sep 26 09:29:45 dist-test-slave-dfzm krb5kdc[12127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:45.725751 13458 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:29:45.726255 13458 webserver.cc:460] Webserver started at http://127.0.98.3:38495/ using document root <none> and password file <none>
I20260926 09:29:45.726493 13458 fs_manager.cc:359] Metadata directory not provided
I20260926 09:29:45.726575 13458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 09:29:45 dist-test-slave-dfzm krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:29:45.748402 13458 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:29:45.749213 13474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:45.749424 13458 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:29:45.749503 13458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "75a0582140694938800476f012de866f"
format_stamp: "Formatted at 2026-09-26 09:29:44 on dist-test-slave-dfzm"
server_key: "8a7a129682e524ba540f246d84811cfb"
server_key_iv: "e2ff70dd5cf0a3a5906c11741da7a06e"
server_key_version: "kuduclusterkey@0"
I20260926 09:29:45.749980 13458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:29:45.775485 13458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:29:45.777688 13458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:29:45.777841 13458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:29:45.778079 13458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:29:45.778421 13458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:29:45.778479 13458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:45.778522 13458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:29:45.778548 13458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:45.785614 13458 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36397
I20260926 09:29:45.787176 13458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:29:45.787555 13588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36397 every 8 connection(s)
I20260926 09:29:45.792157 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 13458
Sep 26 09:29:45 dist-test-slave-dfzm krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:29:45.799213 13324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:45.788477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42163 (local address 127.0.98.62:36741)
0926 09:29:45.788585 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:45.788587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:45.788599 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:29:45.789109 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:45.789110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:45.789123 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:45.789225 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:45.789913 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:45.790608 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:45.792217 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:45.792541 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:45.792619 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:45.795353 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:45.795355 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:45.795355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:45.795364 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:45.797231 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:45.797851 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:45.797853 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:45.797854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:45.797925 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:45.798310 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:45.798312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:45.798312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:45.798459 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:45.798526 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:45.799019 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 09:29:45.799104 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 09:29:45.799388 13591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:45.788210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36741 (local address 127.0.98.3:42163)
0926 09:29:45.788408 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:45.788446 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:45.788996 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:45.789347 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:45.789351 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:45.789540 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:29:45.789821 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:45.789827 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:45.790725 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:45.790726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:45.792057 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:45.792059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:45.792663 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:45.792665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:45.792817 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:45.793525 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:29:45.793536 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:29:45.795237 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:29:45.797323 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:45.797332 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:45.797335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:45.797733 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:45.798021 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:45.798022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:45.798022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:45.798202 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:45.798610 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:29:45.798617 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:29:45.798811 (+ 194us) client_negotiation.cc:769] Sending connection context
0926 09:29:45.798958 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 09:29:45.799139 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":102,"threads_started":1}
I20260926 09:29:45.799965 13589 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36741
I20260926 09:29:45.800122 13589 heartbeater.cc:464] Registering TS with master...
I20260926 09:29:45.800392 13589 heartbeater.cc:511] Master 127.0.98.62:36741 requested a full tablet report, sending...
W20260926 09:29:45.801429 13589 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36741 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 09:29:45.801561 13589 heartbeater.cc:464] Registering TS with master...
I20260926 09:29:45.801666 13589 heartbeater.cc:511] Master 127.0.98.62:36741 requested a full tablet report, sending...
I20260926 09:29:45.801991 13589 heartbeater.cc:464] Registering TS with master...
I20260926 09:29:45.802080 13589 heartbeater.cc:511] Master 127.0.98.62:36741 requested a full tablet report, sending...
W20260926 09:29:45.802379 13589 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 09:29:45.806989 13593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:45.792790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60836 (local address 127.0.98.62:36741)
0926 09:29:45.792951 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:45.792953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:45.793001 (+ 48us) server_negotiation.cc:407] Connection header received
0926 09:29:45.793021 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:45.793021 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:45.793031 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:45.797773 (+ 4742us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:45.797820 (+ 47us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:45.798533 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:45.800667 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:45.801150 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:45.801243 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:45.803055 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:45.803056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:45.803057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:45.803064 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:45.804824 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:45.806144 (+ 1320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:45.806145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:45.806145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:45.806209 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:45.806441 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:45.806442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:45.806442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:45.806551 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:45.806737 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:45.806784 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 09:29:45.806864 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":60,"threads_started":1}
Sep 26 09:29:46 dist-test-slave-dfzm krb5kdc[12127](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790414986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:46.421689 13321 heartbeater.cc:464] Registering TS with master...
I20260926 09:29:46.421870 13321 heartbeater.cc:511] Master 127.0.98.62:36741 requested a full tablet report, sending...
I20260926 09:29:46.422243 13123 ts_manager.cc:194] Registered new tserver with Master: 02be29d50ad8404381782363b3f4817a (127.0.98.1:34139)
I20260926 09:29:46.422947 13123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38057
I20260926 09:29:46.623268 13455 heartbeater.cc:464] Registering TS with master...
I20260926 09:29:46.623452 13455 heartbeater.cc:511] Master 127.0.98.62:36741 requested a full tablet report, sending...
I20260926 09:29:46.623821 13123 ts_manager.cc:194] Registered new tserver with Master: 29b964e356c24c7ca5022796c8abc01b (127.0.98.2:42095)
I20260926 09:29:46.624377 13123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35745
Sep 26 09:29:46 dist-test-slave-dfzm krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:29:46.802601 13589 heartbeater.cc:464] Registering TS with master...
I20260926 09:29:46.802762 13589 heartbeater.cc:511] Master 127.0.98.62:36741 requested a full tablet report, sending...
I20260926 09:29:46.803099 13123 ts_manager.cc:194] Registered new tserver with Master: 75a0582140694938800476f012de866f (127.0.98.3:36397)
I20260926 09:29:46.803625 13123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42163
I20260926 09:29:46.812000 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:29:46.831482 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:46.813104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60842 (local address 127.0.98.62:36741)
0926 09:29:46.813207 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:46.813209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:46.823607 (+ 10398us) server_negotiation.cc:407] Connection header received
0926 09:29:46.823657 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:46.823658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:46.823674 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:46.823769 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:46.824566 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:46.825365 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:46.826443 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:46.826761 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:46.826826 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:46.828335 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:46.828337 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:46.828337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:46.828345 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:46.830195 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:46.830598 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:46.830599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:46.830599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:46.830664 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:46.830890 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:46.830891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:46.830892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:46.831037 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:46.831136 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:46.831291 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 09:29:46.831376 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":53,"threads_started":1}
I20260926 09:29:46.833971 13123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:29:46.836411 13123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:29:46.845137 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:46.839928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45408 (local address 127.0.98.3:36397)
0926 09:29:46.840068 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:46.840074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:46.840538 (+ 464us) server_negotiation.cc:407] Connection header received
0926 09:29:46.840645 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:46.840648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:46.840664 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:46.840742 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:29:46.840988 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:46.842074 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:46.843783 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:46.844553 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:46.844629 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:46.844705 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:46.845003 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 09:29:46.845051 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260926 09:29:46.845443 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:46.839498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34139 (local address 127.0.0.1:56004)
0926 09:29:46.840677 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:46.840685 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:46.840724 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:46.840967 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:46.840967 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:46.840969 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:29:46.841017 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:46.841018 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:46.842458 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:46.842459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:46.843667 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:46.843670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:46.845162 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:46.845163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:46.845224 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:46.845228 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:29:46.845282 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 09:29:46.845360 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1164,"thread_start_us":86,"threads_started":1}
I20260926 09:29:46.845678 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:46.839488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36397 (local address 127.0.0.1:45408)
0926 09:29:46.840437 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:46.840478 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:46.840553 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:46.840827 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:46.840831 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:46.840833 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:29:46.840900 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:46.840901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:46.842329 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:46.842330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:46.843645 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:46.843648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:46.844742 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:46.844743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:46.844855 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:46.844869 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 09:29:46.845527 (+ 658us) client_negotiation.cc:240] Negotiation successful
0926 09:29:46.845574 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":4,"thread_start_us":79,"threads_started":1}
I20260926 09:29:46.846256 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:46.840079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56004 (local address 127.0.98.1:34139)
0926 09:29:46.840227 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:46.840231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:46.840776 (+ 545us) server_negotiation.cc:407] Connection header received
0926 09:29:46.840812 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:46.840815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:46.840830 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:46.840894 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:29:46.841083 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:46.842163 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:46.843769 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:46.844920 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:46.845048 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:46.845133 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:46.846135 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0926 09:29:46.846179 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":69,"threads_started":1}
I20260926 09:29:46.846566 13523 tablet_service.cc:1467] Processing CreateTablet for tablet cedc22f4c6be49b68474a14e2fb34bc9 (DEFAULT_TABLE table=test-table [id=22012468aa2f49328f414203f45f80bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:29:46.846946 13523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cedc22f4c6be49b68474a14e2fb34bc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:29:46.847057 13255 tablet_service.cc:1467] Processing CreateTablet for tablet cedc22f4c6be49b68474a14e2fb34bc9 (DEFAULT_TABLE table=test-table [id=22012468aa2f49328f414203f45f80bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:29:46.847412 13255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cedc22f4c6be49b68474a14e2fb34bc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:29:46.850422 13623 tablet_bootstrap.cc:492] T cedc22f4c6be49b68474a14e2fb34bc9 P 02be29d50ad8404381782363b3f4817a: Bootstrap starting.
I20260926 09:29:46.851575 13623 tablet_bootstrap.cc:654] T cedc22f4c6be49b68474a14e2fb34bc9 P 02be29d50ad8404381782363b3f4817a: Neither blocks nor log segments found. Creating new log.
I20260926 09:29:46.851894 13623 log.cc:824] T cedc22f4c6be49b68474a14e2fb34bc9 P 02be29d50ad8404381782363b3f4817a: Log is configured to *not* fsync() on all Append() calls
I20260926 09:29:46.852231 13624 tablet_bootstrap.cc:492] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f: Bootstrap starting.
I20260926 09:29:46.852717 13623 tablet_bootstrap.cc:492] T cedc22f4c6be49b68474a14e2fb34bc9 P 02be29d50ad8404381782363b3f4817a: No bootstrap required, opened a new log
I20260926 09:29:46.852787 13623 ts_tablet_manager.cc:1397] T cedc22f4c6be49b68474a14e2fb34bc9 P 02be29d50ad8404381782363b3f4817a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:29:46.855644 13623 raft_consensus.cc:348] T cedc22f4c6be49b68474a14e2fb34bc9 P 02be29d50ad8404381782363b3f4817a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75a0582140694938800476f012de866f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36397 } } peers { permanent_uuid: "02be29d50ad8404381782363b3f4817a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34139 } } peers { permanent_uuid: "29b964e356c24c7ca5022796c8abc01b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42095 } }
I20260926 09:29:46.855931 13623 raft_consensus.cc:374] T cedc22f4c6be49b68474a14e2fb34bc9 P 02be29d50ad8404381782363b3f4817a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:29:46.856030 13623 raft_consensus.cc:729] T cedc22f4c6be49b68474a14e2fb34bc9 P 02be29d50ad8404381782363b3f4817a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02be29d50ad8404381782363b3f4817a, State: Initialized, Role: FOLLOWER
I20260926 09:29:46.856248 13623 consensus_queue.cc:260] T cedc22f4c6be49b68474a14e2fb34bc9 P 02be29d50ad8404381782363b3f4817a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75a0582140694938800476f012de866f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36397 } } peers { permanent_uuid: "02be29d50ad8404381782363b3f4817a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34139 } } peers { permanent_uuid: "29b964e356c24c7ca5022796c8abc01b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42095 } }
I20260926 09:29:46.856652 13623 ts_tablet_manager.cc:1428] T cedc22f4c6be49b68474a14e2fb34bc9 P 02be29d50ad8404381782363b3f4817a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 09:29:46.856817 13321 heartbeater.cc:503] Master 127.0.98.62:36741 was elected leader, sending a full tablet report...
I20260926 09:29:46.857672 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:46.839717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42095 (local address 127.0.0.1:37274)
0926 09:29:46.845645 (+ 5928us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:46.845655 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:46.845711 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:46.853937 (+ 8226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:46.853938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:46.853941 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:29:46.854002 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:46.854004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:46.855414 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:46.855415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:46.856506 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:46.856509 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:46.857440 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:46.857442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:46.857502 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:46.857507 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:29:46.857536 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 09:29:46.857579 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5910,"thread_start_us":68,"threads_started":1}
I20260926 09:29:46.858060 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:46.839813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37274 (local address 127.0.98.2:42095)
0926 09:29:46.853757 (+ 13944us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:46.853762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:46.853775 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:29:46.853811 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:46.853812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:46.853829 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:46.854080 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:29:46.854161 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:46.855331 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:46.856669 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:46.857347 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:46.857831 (+ 484us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:46.857918 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:46.857944 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 09:29:46.857992 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13902,"thread_start_us":55,"threads_started":1}
I20260926 09:29:46.858767 13388 tablet_service.cc:1467] Processing CreateTablet for tablet cedc22f4c6be49b68474a14e2fb34bc9 (DEFAULT_TABLE table=test-table [id=22012468aa2f49328f414203f45f80bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:29:46.859138 13388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cedc22f4c6be49b68474a14e2fb34bc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:29:46.862023 13624 tablet_bootstrap.cc:654] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f: Neither blocks nor log segments found. Creating new log.
I20260926 09:29:46.862820 13624 log.cc:824] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f: Log is configured to *not* fsync() on all Append() calls
I20260926 09:29:46.863679 13627 tablet_bootstrap.cc:492] T cedc22f4c6be49b68474a14e2fb34bc9 P 29b964e356c24c7ca5022796c8abc01b: Bootstrap starting.
I20260926 09:29:46.863708 13624 tablet_bootstrap.cc:492] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f: No bootstrap required, opened a new log
I20260926 09:29:46.863770 13624 ts_tablet_manager.cc:1397] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260926 09:29:46.864370 13627 tablet_bootstrap.cc:654] T cedc22f4c6be49b68474a14e2fb34bc9 P 29b964e356c24c7ca5022796c8abc01b: Neither blocks nor log segments found. Creating new log.
I20260926 09:29:46.864650 13627 log.cc:824] T cedc22f4c6be49b68474a14e2fb34bc9 P 29b964e356c24c7ca5022796c8abc01b: Log is configured to *not* fsync() on all Append() calls
I20260926 09:29:46.865368 13627 tablet_bootstrap.cc:492] T cedc22f4c6be49b68474a14e2fb34bc9 P 29b964e356c24c7ca5022796c8abc01b: No bootstrap required, opened a new log
I20260926 09:29:46.865425 13627 ts_tablet_manager.cc:1397] T cedc22f4c6be49b68474a14e2fb34bc9 P 29b964e356c24c7ca5022796c8abc01b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:29:46.866328 13624 raft_consensus.cc:348] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75a0582140694938800476f012de866f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36397 } } peers { permanent_uuid: "02be29d50ad8404381782363b3f4817a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34139 } } peers { permanent_uuid: "29b964e356c24c7ca5022796c8abc01b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42095 } }
I20260926 09:29:46.866484 13624 raft_consensus.cc:374] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:29:46.866519 13624 raft_consensus.cc:729] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a0582140694938800476f012de866f, State: Initialized, Role: FOLLOWER
I20260926 09:29:46.866628 13624 consensus_queue.cc:260] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75a0582140694938800476f012de866f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36397 } } peers { permanent_uuid: "02be29d50ad8404381782363b3f4817a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34139 } } peers { permanent_uuid: "29b964e356c24c7ca5022796c8abc01b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42095 } }
I20260926 09:29:46.866861 13624 ts_tablet_manager.cc:1428] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:29:46.866897 13589 heartbeater.cc:503] Master 127.0.98.62:36741 was elected leader, sending a full tablet report...
I20260926 09:29:46.868921 13627 raft_consensus.cc:348] T cedc22f4c6be49b68474a14e2fb34bc9 P 29b964e356c24c7ca5022796c8abc01b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75a0582140694938800476f012de866f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36397 } } peers { permanent_uuid: "02be29d50ad8404381782363b3f4817a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34139 } } peers { permanent_uuid: "29b964e356c24c7ca5022796c8abc01b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42095 } }
I20260926 09:29:46.869083 13627 raft_consensus.cc:374] T cedc22f4c6be49b68474a14e2fb34bc9 P 29b964e356c24c7ca5022796c8abc01b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:29:46.869145 13627 raft_consensus.cc:729] T cedc22f4c6be49b68474a14e2fb34bc9 P 29b964e356c24c7ca5022796c8abc01b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b964e356c24c7ca5022796c8abc01b, State: Initialized, Role: FOLLOWER
I20260926 09:29:46.869268 13627 consensus_queue.cc:260] T cedc22f4c6be49b68474a14e2fb34bc9 P 29b964e356c24c7ca5022796c8abc01b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75a0582140694938800476f012de866f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36397 } } peers { permanent_uuid: "02be29d50ad8404381782363b3f4817a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34139 } } peers { permanent_uuid: "29b964e356c24c7ca5022796c8abc01b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42095 } }
I20260926 09:29:46.869539 13455 heartbeater.cc:503] Master 127.0.98.62:36741 was elected leader, sending a full tablet report...
I20260926 09:29:46.869536 13627 ts_tablet_manager.cc:1428] T cedc22f4c6be49b68474a14e2fb34bc9 P 29b964e356c24c7ca5022796c8abc01b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
W20260926 09:29:46.904918 13322 tablet.cc:2369] T cedc22f4c6be49b68474a14e2fb34bc9 P 02be29d50ad8404381782363b3f4817a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:29:46.978447 13630 raft_consensus.cc:482] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:29:46.978605 13630 raft_consensus.cc:504] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75a0582140694938800476f012de866f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36397 } } peers { permanent_uuid: "02be29d50ad8404381782363b3f4817a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34139 } } peers { permanent_uuid: "29b964e356c24c7ca5022796c8abc01b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42095 } }
I20260926 09:29:46.979018 13630 leader_election.cc:288] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02be29d50ad8404381782363b3f4817a (127.0.98.1:34139), 29b964e356c24c7ca5022796c8abc01b (127.0.98.2:42095)
Sep 26 09:29:46 dist-test-slave-dfzm krb5kdc[12127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 09:29:46.990110 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:46.979664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39851 (local address 127.0.98.1:34139)
0926 09:29:46.979775 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:46.979777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:46.980775 (+ 998us) server_negotiation.cc:407] Connection header received
0926 09:29:46.980890 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:46.980892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:46.980905 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:46.980950 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:29:46.981494 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:46.982279 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:46.985719 (+ 3440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:46.986333 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:46.986380 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:46.986392 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:46.989971 (+ 3579us) server_negotiation.cc:299] Negotiation successful
0926 09:29:46.990031 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 09:29:46.990350 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:46.979220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34139 (local address 127.0.98.3:39851)
0926 09:29:46.980704 (+ 1484us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:46.980717 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:46.980808 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:46.981288 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:46.981290 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:46.981292 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:29:46.981390 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:46.981392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:46.982524 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:46.982525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:46.984632 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:46.984635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:46.989789 (+ 5154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:46.989790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:46.989875 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:46.989880 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:29:46.990199 (+ 319us) client_negotiation.cc:240] Negotiation successful
0926 09:29:46.990253 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1462,"thread_start_us":105,"threads_started":1}
I20260926 09:29:46.992817 13273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cedc22f4c6be49b68474a14e2fb34bc9" candidate_uuid: "75a0582140694938800476f012de866f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02be29d50ad8404381782363b3f4817a" is_pre_election: true
I20260926 09:29:46.993010 13273 raft_consensus.cc:2454] T cedc22f4c6be49b68474a14e2fb34bc9 P 02be29d50ad8404381782363b3f4817a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75a0582140694938800476f012de866f in term 0.
I20260926 09:29:46.993516 13475 leader_election.cc:302] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02be29d50ad8404381782363b3f4817a, 75a0582140694938800476f012de866f; no voters:
I20260926 09:29:46.993882 13630 raft_consensus.cc:2781] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:29:46.993957 13630 raft_consensus.cc:482] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:29:46.993984 13630 raft_consensus.cc:3037] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:29:46.994925 13630 raft_consensus.cc:504] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75a0582140694938800476f012de866f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36397 } } peers { permanent_uuid: "02be29d50ad8404381782363b3f4817a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34139 } } peers { permanent_uuid: "29b964e356c24c7ca5022796c8abc01b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42095 } }
I20260926 09:29:46.995060 13630 leader_election.cc:288] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f [CANDIDATE]: Term 1 election: Requested vote from peers 02be29d50ad8404381782363b3f4817a (127.0.98.1:34139), 29b964e356c24c7ca5022796c8abc01b (127.0.98.2:42095)
I20260926 09:29:46.995385 13273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cedc22f4c6be49b68474a14e2fb34bc9" candidate_uuid: "75a0582140694938800476f012de866f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02be29d50ad8404381782363b3f4817a"
I20260926 09:29:46.995476 13273 raft_consensus.cc:3037] T cedc22f4c6be49b68474a14e2fb34bc9 P 02be29d50ad8404381782363b3f4817a [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:29:46.996358 13273 raft_consensus.cc:2454] T cedc22f4c6be49b68474a14e2fb34bc9 P 02be29d50ad8404381782363b3f4817a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75a0582140694938800476f012de866f in term 1.
I20260926 09:29:46.996615 13475 leader_election.cc:302] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02be29d50ad8404381782363b3f4817a, 75a0582140694938800476f012de866f; no voters:
I20260926 09:29:46.996732 13630 raft_consensus.cc:2781] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:29:46.996896 13630 raft_consensus.cc:686] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f [term 1 LEADER]: Becoming Leader. State: Replica: 75a0582140694938800476f012de866f, State: Running, Role: LEADER
I20260926 09:29:46.996984 13630 consensus_queue.cc:237] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75a0582140694938800476f012de866f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36397 } } peers { permanent_uuid: "02be29d50ad8404381782363b3f4817a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34139 } } peers { permanent_uuid: "29b964e356c24c7ca5022796c8abc01b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42095 } }
I20260926 09:29:46.997916 13123 catalog_manager.cc:5495] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f reported cstate change: term changed from 0 to 1, leader changed from <none> to 75a0582140694938800476f012de866f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "75a0582140694938800476f012de866f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75a0582140694938800476f012de866f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02be29d50ad8404381782363b3f4817a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29b964e356c24c7ca5022796c8abc01b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42095 } health_report { overall_health: UNKNOWN } } }
I20260926 09:29:46.999840 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:46.979726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53163 (local address 127.0.98.2:42095)
0926 09:29:46.979787 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:46.979790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:46.989800 (+ 10010us) server_negotiation.cc:407] Connection header received
0926 09:29:46.989910 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:46.989911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:46.989924 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:46.989954 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:29:46.990177 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:46.990862 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:46.998691 (+ 7829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:46.999349 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:46.999401 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:46.999413 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:46.999724 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 09:29:46.999770 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 09:29:47.000288 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:46.979445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42095 (local address 127.0.98.3:53163)
0926 09:29:46.989716 (+ 10271us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:46.989727 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:46.989833 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:46.990032 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:46.990033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:46.990035 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:29:46.990103 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:46.990105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:46.990989 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:46.990991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:46.992151 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:46.992154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:46.999564 (+ 7410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:46.999565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:46.999642 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:46.999646 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:29:47.000162 (+ 516us) client_negotiation.cc:240] Negotiation successful
0926 09:29:47.000214 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10247,"thread_start_us":69,"threads_started":1}
I20260926 09:29:47.000717 13409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cedc22f4c6be49b68474a14e2fb34bc9" candidate_uuid: "75a0582140694938800476f012de866f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b964e356c24c7ca5022796c8abc01b" is_pre_election: true
I20260926 09:29:47.000859 13409 raft_consensus.cc:2454] T cedc22f4c6be49b68474a14e2fb34bc9 P 29b964e356c24c7ca5022796c8abc01b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75a0582140694938800476f012de866f in term 0.
I20260926 09:29:47.000954 13408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cedc22f4c6be49b68474a14e2fb34bc9" candidate_uuid: "75a0582140694938800476f012de866f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b964e356c24c7ca5022796c8abc01b"
I20260926 09:29:47.001014 13408 raft_consensus.cc:3037] T cedc22f4c6be49b68474a14e2fb34bc9 P 29b964e356c24c7ca5022796c8abc01b [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:29:47.001840 13408 raft_consensus.cc:2454] T cedc22f4c6be49b68474a14e2fb34bc9 P 29b964e356c24c7ca5022796c8abc01b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75a0582140694938800476f012de866f in term 1.
I20260926 09:29:47.026130 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:47.022719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45414 (local address 127.0.98.3:36397)
0926 09:29:47.022769 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:47.022771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:47.022779 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:29:47.022825 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:47.022826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:47.022853 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:47.022951 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:29:47.023112 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:47.023805 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:47.024890 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:47.025164 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:47.025233 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:47.025462 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:29:47.025659 (+ 197us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:29:47.025750 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:47.026009 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 09:29:47.026034 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:29:47.028337 13408 raft_consensus.cc:1261] T cedc22f4c6be49b68474a14e2fb34bc9 P 29b964e356c24c7ca5022796c8abc01b [term 1 FOLLOWER]: Refusing update from remote peer 75a0582140694938800476f012de866f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:29:47.028337 13273 raft_consensus.cc:1261] T cedc22f4c6be49b68474a14e2fb34bc9 P 02be29d50ad8404381782363b3f4817a [term 1 FOLLOWER]: Refusing update from remote peer 75a0582140694938800476f012de866f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:29:47.028714 13630 consensus_queue.cc:1035] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b964e356c24c7ca5022796c8abc01b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:29:47.028893 13630 consensus_queue.cc:1035] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f [LEADER]: Connected to new peer: Peer: permanent_uuid: "02be29d50ad8404381782363b3f4817a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:29:47.033063 13638 mvcc.cc:205] Tried to move back new op lower bound from 7333539786864431104 to 7333539786740027392. Current Snapshot: MvccSnapshot[applied={T|T < 7333539786864431104}]
I20260926 09:29:47.034205 13640 mvcc.cc:205] Tried to move back new op lower bound from 7333539786864431104 to 7333539786740027392. Current Snapshot: MvccSnapshot[applied={T|T < 7333539786864431104}]
I20260926 09:29:47.039512 13122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:29:47.039630 13122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:29:47.042562 13122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f963fdd4a465452295c73563e4967760: Sending DeleteTablet for 3 replicas of tablet cedc22f4c6be49b68474a14e2fb34bc9
I20260926 09:29:47.043126 13523 tablet_service.cc:1514] Processing DeleteTablet for tablet cedc22f4c6be49b68474a14e2fb34bc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:29:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45408
I20260926 09:29:47.043396 13388 tablet_service.cc:1514] Processing DeleteTablet for tablet cedc22f4c6be49b68474a14e2fb34bc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:29:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37274
I20260926 09:29:47.043960 13255 tablet_service.cc:1514] Processing DeleteTablet for tablet cedc22f4c6be49b68474a14e2fb34bc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:29:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56004
I20260926 09:29:47.044070 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 13161
I20260926 09:29:47.047430 13648 tablet_replica.cc:330] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f: stopping tablet replica
I20260926 09:29:47.047549 13648 raft_consensus.cc:2229] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:29:47.047648 13648 raft_consensus.cc:2258] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:29:47.047820 13649 tablet_replica.cc:330] T cedc22f4c6be49b68474a14e2fb34bc9 P 29b964e356c24c7ca5022796c8abc01b: stopping tablet replica
I20260926 09:29:47.047927 13649 raft_consensus.cc:2229] T cedc22f4c6be49b68474a14e2fb34bc9 P 29b964e356c24c7ca5022796c8abc01b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:29:47.048054 13649 raft_consensus.cc:2258] T cedc22f4c6be49b68474a14e2fb34bc9 P 29b964e356c24c7ca5022796c8abc01b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:29:47.048565 13648 ts_tablet_manager.cc:1905] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:29:47.049023 13649 ts_tablet_manager.cc:1905] T cedc22f4c6be49b68474a14e2fb34bc9 P 29b964e356c24c7ca5022796c8abc01b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:29:47.050302 13648 ts_tablet_manager.cc:1918] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:29:47.050473 13648 log.cc:1196] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/cedc22f4c6be49b68474a14e2fb34bc9
I20260926 09:29:47.050835 13648 ts_tablet_manager.cc:1939] T cedc22f4c6be49b68474a14e2fb34bc9 P 75a0582140694938800476f012de866f: Deleting consensus metadata
I20260926 09:29:47.051330 13109 catalog_manager.cc:4841] TS 75a0582140694938800476f012de866f (127.0.98.3:36397): tablet cedc22f4c6be49b68474a14e2fb34bc9 (table test-table [id=22012468aa2f49328f414203f45f80bc]) successfully deleted
I20260926 09:29:47.066183 13649 ts_tablet_manager.cc:1918] T cedc22f4c6be49b68474a14e2fb34bc9 P 29b964e356c24c7ca5022796c8abc01b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:29:47.066439 13649 log.cc:1196] T cedc22f4c6be49b68474a14e2fb34bc9 P 29b964e356c24c7ca5022796c8abc01b: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414839069479-392-0/minicluster-data/ts-1/wal/wals/cedc22f4c6be49b68474a14e2fb34bc9
I20260926 09:29:47.066808 13649 ts_tablet_manager.cc:1939] T cedc22f4c6be49b68474a14e2fb34bc9 P 29b964e356c24c7ca5022796c8abc01b: Deleting consensus metadata
I20260926 09:29:47.067395 13107 catalog_manager.cc:4841] TS 29b964e356c24c7ca5022796c8abc01b (127.0.98.2:42095): tablet cedc22f4c6be49b68474a14e2fb34bc9 (table test-table [id=22012468aa2f49328f414203f45f80bc]) successfully deleted
W20260926 09:29:47.069965 13108 connection.cc:535] server connection from 127.0.98.1:38057 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 09:29:47.070233 13107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34139 (error 108)
I20260926 09:29:47.071125 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:47.071006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34139 (local address 127.0.0.1:56006)
0926 09:29:47.071059 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:47.071090 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 09:29:47.071357 13107 catalog_manager.cc:4584] TS 02be29d50ad8404381782363b3f4817a (127.0.98.1:34139): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cedc22f4c6be49b68474a14e2fb34bc9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34139: connect: Connection refused (error 111)
I20260926 09:29:47.071628 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 13325
I20260926 09:29:47.078164 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 13458
I20260926 09:29:47.084877 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 13090
I20260926 09:29:47.091933 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 09:29:47.437325 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 09:29:47.437430 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 09:29:47.782059 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 09:29:47.782 UTC [12157] LOG: received smart shutdown request
2026-09-26 09:29:47.786 UTC [12157] LOG: background worker "logical replication launcher" (PID 12166) exited with exit code 1
2026-09-26 09:29:47.786 UTC [12161] LOG: shutting down
2026-09-26 09:29:47.956 UTC [12157] LOG: database system is shut down
2026-09-26T09:29:47Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47907 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:29:48 dist-test-slave-dfzm krb5kdc[13660](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:29:48 dist-test-slave-dfzm krb5kdc[13660](info): set up 2 sockets
Sep 26 09:29:48 dist-test-slave-dfzm krb5kdc[13660](info): commencing operation
krb5kdc: starting...
W20260926 09:29:49.117511 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:29:49 dist-test-slave-dfzm krb5kdc[13660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:29:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:29:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:29:49.240895 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44405
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44405,127.0.98.61:36505,127.0.98.60:45727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:29:49.323815 13673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:29:49.324082 13673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:29:49.324123 13673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:29:49.325321 13673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:29:49.325402 13673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:29:49.325428 13673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:29:49.325460 13673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:29:49.325493 13673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:29:49.327154 13673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44405,127.0.98.61:36505,127.0.98.60:45727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:29:49.327451 13673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:29:49.327713 13673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:29:49.332878 13678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:49.332896 13681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:49.332878 13679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:49.333199 13673 server_base.cc:1034] running on GCE node
I20260926 09:29:49.333482 13673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:29:49.333827 13673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:29:49.335000 13673 hybrid_clock.cc:648] HybridClock initialized: now 1790414989334986 us; error 27 us; skew 500 ppm
Sep 26 09:29:49 dist-test-slave-dfzm krb5kdc[13660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:49.338164 13673 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:29:49.338634 13673 webserver.cc:460] Webserver started at http://127.0.98.62:39415/ using document root <none> and password file <none>
I20260926 09:29:49.338891 13673 fs_manager.cc:359] Metadata directory not provided
I20260926 09:29:49.338938 13673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:29:49.339094 13673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:29:49.340214 13673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "fa9295dd63cf4ba78255db4aee6151b0"
format_stamp: "Formatted at 2026-09-26 09:29:49 on dist-test-slave-dfzm"
server_key: "52e08ae76c09c3138ce6a4e3f8c7afed"
server_key_iv: "d075a7a3d83ac8ce0c0a31f4a9f013f9"
server_key_version: "encryptionkey@0"
I20260926 09:29:49.340585 13673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "fa9295dd63cf4ba78255db4aee6151b0"
format_stamp: "Formatted at 2026-09-26 09:29:49 on dist-test-slave-dfzm"
server_key: "52e08ae76c09c3138ce6a4e3f8c7afed"
server_key_iv: "d075a7a3d83ac8ce0c0a31f4a9f013f9"
server_key_version: "encryptionkey@0"
I20260926 09:29:49.342010 13673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:29:49.342765 13688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:49.342995 13673 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:29:49.343072 13673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "fa9295dd63cf4ba78255db4aee6151b0"
format_stamp: "Formatted at 2026-09-26 09:29:49 on dist-test-slave-dfzm"
server_key: "52e08ae76c09c3138ce6a4e3f8c7afed"
server_key_iv: "d075a7a3d83ac8ce0c0a31f4a9f013f9"
server_key_version: "encryptionkey@0"
I20260926 09:29:49.343166 13673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:29:49.357242 13673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:29:49.359344 13673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:29:49.359506 13673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:29:49.363561 13673 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44405
I20260926 09:29:49.363618 13740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44405 every 8 connection(s)
I20260926 09:29:49.364009 13673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:29:49.364887 13741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:29:49.366381 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 13673
I20260926 09:29:49.366506 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:29:49.366680 392 external_mini_cluster.cc:1428] Setting key 78caa0cd4623e939a6cc8ec9d2ed85c7
I20260926 09:29:49.366778 13741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:29:49 dist-test-slave-dfzm krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260926 09:29:49.377811 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.367642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44405 (local address 127.0.0.1:56928)
0926 09:29:49.367900 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.367942 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:49.368438 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:49.368727 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:49.368730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:49.369030 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:29:49.369184 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:49.369189 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.370500 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.370501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:49.371268 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:49.371271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.371842 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.371843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:49.371915 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:49.372592 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:29:49.372600 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:29:49.374279 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:29:49.376160 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:49.376167 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:49.376169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:49.376435 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:49.376700 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:49.376701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:49.376702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:49.376796 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:49.377177 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:29:49.377182 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:29:49.377243 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 09:29:49.377360 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 09:29:49.377621 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":89,"thread_start_us":105,"threads_started":1}
I20260926 09:29:49.377811 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.367721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.0.98.62:44405)
0926 09:29:49.367898 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:49.367908 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:49.368040 (+ 132us) server_negotiation.cc:407] Connection header received
0926 09:29:49.368608 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:49.368610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:49.368626 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:49.368723 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:49.369344 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.370260 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.371388 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.371735 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.371874 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:49.374366 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:49.374373 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:49.374373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:49.374379 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:49.376014 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.376524 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.376527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.376528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.376626 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.376881 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.376882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.376882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.377104 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:49.377199 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:49.377383 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 09:29:49.377528 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260926 09:29:49.378610 13741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:29:49.378978 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.378860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36505 (local address 127.0.0.1:42714)
0926 09:29:49.378907 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.378950 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 09:29:49.379146 13692 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36505: connect: Connection refused (error 111)
I20260926 09:29:49.379592 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.379495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36505 (local address 127.0.0.1:42722)
0926 09:29:49.379545 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.379575 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 09:29:49.379719 13741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36505: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36505: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 09:29:49.393102 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36505
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44405,127.0.98.61:36505,127.0.98.60:45727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:29:49.428954 13741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } attempt: 1
I20260926 09:29:49.429527 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.429380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36505 (local address 127.0.0.1:42730)
0926 09:29:49.429450 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.429498 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 09:29:49.429953 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.429844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36505 (local address 127.0.0.1:42732)
0926 09:29:49.429899 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.429932 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 09:29:49.430070 13741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36505: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36505: connect: Connection refused (error 111)
W20260926 09:29:49.475317 13748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:29:49.475569 13748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:29:49.475610 13748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:29:49.476828 13748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:29:49.476895 13748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:29:49.476920 13748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:29:49.476974 13748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:29:49.477001 13748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:29:49.478645 13748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44405,127.0.98.61:36505,127.0.98.60:45727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:29:49.478919 13748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:29:49.479142 13748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:29:49.484090 13754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:49.484136 13753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:49.484169 13748 server_base.cc:1034] running on GCE node
W20260926 09:29:49.484136 13756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:49.484524 13748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:29:49.484874 13748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:29:49.486022 13748 hybrid_clock.cc:648] HybridClock initialized: now 1790414989486002 us; error 34 us; skew 500 ppm
Sep 26 09:29:49 dist-test-slave-dfzm krb5kdc[13660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:49.488559 13748 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 09:29:49.489039 13748 webserver.cc:460] Webserver started at http://127.0.98.61:37845/ using document root <none> and password file <none>
I20260926 09:29:49.489275 13748 fs_manager.cc:359] Metadata directory not provided
I20260926 09:29:49.489347 13748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:29:49.489504 13748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:29:49.490617 13748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-1/data/instance:
uuid: "9954aab5698441c399f3b16df9857a35"
format_stamp: "Formatted at 2026-09-26 09:29:49 on dist-test-slave-dfzm"
server_key: "c868b99d3552cf26210c68ebe6ff9043"
server_key_iv: "9e2f9393e1357c69bc887572927e00b7"
server_key_version: "encryptionkey@0"
I20260926 09:29:49.490993 13748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-1/wal/instance:
uuid: "9954aab5698441c399f3b16df9857a35"
format_stamp: "Formatted at 2026-09-26 09:29:49 on dist-test-slave-dfzm"
server_key: "c868b99d3552cf26210c68ebe6ff9043"
server_key_iv: "9e2f9393e1357c69bc887572927e00b7"
server_key_version: "encryptionkey@0"
I20260926 09:29:49.492334 13748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:29:49.493073 13763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:49.493239 13748 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:29:49.493359 13748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-1/wal
uuid: "9954aab5698441c399f3b16df9857a35"
format_stamp: "Formatted at 2026-09-26 09:29:49 on dist-test-slave-dfzm"
server_key: "c868b99d3552cf26210c68ebe6ff9043"
server_key_iv: "9e2f9393e1357c69bc887572927e00b7"
server_key_version: "encryptionkey@0"
I20260926 09:29:49.493448 13748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:29:49.498215 13741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } attempt: 2
I20260926 09:29:49.498622 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.498479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36505 (local address 127.0.0.1:42742)
0926 09:29:49.498540 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.498587 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 09:29:49.498992 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.498910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36505 (local address 127.0.0.1:42754)
0926 09:29:49.498953 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.498977 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 09:29:49.499110 13741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36505: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36505: connect: Connection refused (error 111)
I20260926 09:29:49.501252 13748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:29:49.503587 13748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:29:49.503750 13748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:29:49.507840 13748 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36505
I20260926 09:29:49.508220 13748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-1/data/info.pb
I20260926 09:29:49.508242 13815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36505 every 8 connection(s)
I20260926 09:29:49.508898 13816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:29:49.510607 13816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:29:49 dist-test-slave-dfzm krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:29:49.517953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 13748
I20260926 09:29:49.518100 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-1/wal/instance
I20260926 09:29:49.518362 392 external_mini_cluster.cc:1428] Setting key e24293b71f78e50c0b2642c1ccd5ba69
I20260926 09:29:49.519891 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.511305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56930 (local address 127.0.98.62:44405)
0926 09:29:49.511384 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:49.511386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:49.511528 (+ 142us) server_negotiation.cc:407] Connection header received
0926 09:29:49.511976 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:49.511977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:49.511989 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:49.512041 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:49.512647 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.513118 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.514198 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.514411 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.514462 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:49.516565 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:49.516566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:49.516566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:49.516574 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:49.518157 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.518706 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.518707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.518708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.518762 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.519066 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.519067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.519067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.519170 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:49.519223 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:49.519642 (+ 419us) server_negotiation.cc:299] Negotiation successful
0926 09:29:49.519721 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 09:29:49.520098 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.511251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44405 (local address 127.0.0.1:56930)
0926 09:29:49.511413 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.511448 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:49.511891 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:49.512114 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:49.512117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:49.512368 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:29:49.512571 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:49.512575 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.513219 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.513220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:49.514111 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:49.514114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.514503 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.514504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:49.514616 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:49.515086 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:29:49.515094 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:29:49.516471 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:29:49.518260 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:49.518270 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:49.518272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:49.518627 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:49.518833 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:49.518834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:49.518834 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:49.518992 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:49.519234 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:29:49.519241 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:29:49.519426 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 09:29:49.519608 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 09:29:49.519802 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260926 09:29:49.520710 13816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:29:49 dist-test-slave-dfzm krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260926 09:29:49.529134 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.520984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36505 (local address 127.0.0.1:42762)
0926 09:29:49.521033 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.521045 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:49.521089 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:49.521387 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:49.521388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:49.521506 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:29:49.521659 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:49.521662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.523015 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.523016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:49.523523 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:49.523525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.524266 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.524267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:49.524330 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:49.524789 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:29:49.524793 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:29:49.526265 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:29:49.528096 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:49.528097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:49.528097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:49.528262 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:49.528429 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:49.528429 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:49.528430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:49.528505 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:49.528809 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:29:49.528810 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:29:49.528847 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 09:29:49.528898 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 09:29:49.529032 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260926 09:29:49.529299 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.521050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42762 (local address 127.0.98.61:36505)
0926 09:29:49.521225 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:49.521230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:49.521252 (+ 22us) server_negotiation.cc:407] Connection header received
0926 09:29:49.521275 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:49.521277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:49.521292 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:49.521375 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:49.521733 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.522922 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.523632 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.524172 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.524259 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:49.526355 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:49.526364 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:49.526364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:49.526373 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:49.528023 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.528328 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.528331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.528331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.528378 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.528584 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.528585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.528585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.528750 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:49.528816 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:49.528939 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 09:29:49.529112 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":143,"thread_start_us":101,"threads_started":1}
I20260926 09:29:49.530154 13816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:29:49.530517 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.530381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45727 (local address 127.0.0.1:41658)
0926 09:29:49.530433 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.530484 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 09:29:49.530715 13765 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45727: connect: Connection refused (error 111)
I20260926 09:29:49.531267 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.531165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45727 (local address 127.0.0.1:41666)
0926 09:29:49.531211 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.531250 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 09:29:49.531369 13816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45727: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45727: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 09:29:49.544613 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45727
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44405,127.0.98.61:36505,127.0.98.60:45727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:29:49.580622 13816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } attempt: 1
I20260926 09:29:49.581143 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.581006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45727 (local address 127.0.0.1:41676)
0926 09:29:49.581076 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.581115 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 09:29:49.581566 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.581476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45727 (local address 127.0.0.1:41684)
0926 09:29:49.581525 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.581551 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 09:29:49.581713 13816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45727: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45727: connect: Connection refused (error 111)
I20260926 09:29:49.590286 13741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } attempt: 3
Sep 26 09:29:49 dist-test-slave-dfzm krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 09:29:49.597693 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.590616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36505 (local address 127.0.0.1:42774)
0926 09:29:49.590675 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.590688 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:49.590767 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:49.590914 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:49.590915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:49.591076 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:29:49.591142 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:49.591143 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.591838 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.591839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:49.592371 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:49.592373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.592787 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.592788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:49.592866 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:49.593339 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:29:49.593344 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:29:49.594808 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:29:49.596556 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:49.596557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:49.596557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:49.596739 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:49.596994 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:49.596995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:49.596995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:49.597061 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:49.597331 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:29:49.597332 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:29:49.597381 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 09:29:49.597457 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 09:29:49.597565 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 09:29:49.597693 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.590687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42774 (local address 127.0.98.61:36505)
0926 09:29:49.590788 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:49.590790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:49.590800 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:29:49.590828 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:49.590829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:49.590841 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:49.590909 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:49.591228 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.591750 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.592472 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.592683 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.592777 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:49.594900 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:49.594902 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:49.594903 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:49.594913 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:49.596469 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.596836 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.596837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.596837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.596917 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.597123 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.597124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.597124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.597249 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:49.597329 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:49.597496 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 09:29:49.597573 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 09:29:49.598034 13741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:29:49.598387 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.598286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45727 (local address 127.0.0.1:41688)
0926 09:29:49.598334 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.598361 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 09:29:49.598762 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.598667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45727 (local address 127.0.0.1:41690)
0926 09:29:49.598713 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.598737 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 09:29:49.598883 13741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45727: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45727: connect: Connection refused (error 111)
W20260926 09:29:49.627030 13823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:29:49.627246 13823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:29:49.627267 13823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:29:49.628424 13823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:29:49.628468 13823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:29:49.628481 13823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:29:49.628494 13823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:29:49.628506 13823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:29:49.630056 13741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } attempt: 1
I20260926 09:29:49.630132 13823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44405,127.0.98.61:36505,127.0.98.60:45727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:29:49.630674 13823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:29:49.630697 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.630507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45727 (local address 127.0.0.1:41692)
0926 09:29:49.630588 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.630651 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 09:29:49.631148 13823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:29:49.631419 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.631258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45727 (local address 127.0.0.1:41708)
0926 09:29:49.631303 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.631398 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260926 09:29:49.631621 13741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45727: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45727: connect: Connection refused (error 111)
W20260926 09:29:49.636984 13828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:49.637032 13829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:49.636984 13831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:49.637193 13823 server_base.cc:1034] running on GCE node
I20260926 09:29:49.637426 13823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:29:49.637729 13823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:29:49.638897 13823 hybrid_clock.cc:648] HybridClock initialized: now 1790414989638881 us; error 48 us; skew 500 ppm
Sep 26 09:29:49 dist-test-slave-dfzm krb5kdc[13660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:49.641476 13823 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 09:29:49.641968 13823 webserver.cc:460] Webserver started at http://127.0.98.60:34021/ using document root <none> and password file <none>
I20260926 09:29:49.642230 13823 fs_manager.cc:359] Metadata directory not provided
I20260926 09:29:49.642304 13823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:29:49.642452 13823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:29:49.643505 13823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-2/data/instance:
uuid: "4c3295d97eba4ac9aebf6b61f589eeb4"
format_stamp: "Formatted at 2026-09-26 09:29:49 on dist-test-slave-dfzm"
server_key: "ec03fd87db55b49c9ad5cbdb1b0e4609"
server_key_iv: "e269b6c555443e37f2d038b109db5c41"
server_key_version: "encryptionkey@0"
I20260926 09:29:49.643870 13823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-2/wal/instance:
uuid: "4c3295d97eba4ac9aebf6b61f589eeb4"
format_stamp: "Formatted at 2026-09-26 09:29:49 on dist-test-slave-dfzm"
server_key: "ec03fd87db55b49c9ad5cbdb1b0e4609"
server_key_iv: "e269b6c555443e37f2d038b109db5c41"
server_key_version: "encryptionkey@0"
I20260926 09:29:49.645345 13823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:29:49.646221 13838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:49.646401 13823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:29:49.646502 13823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-2/wal
uuid: "4c3295d97eba4ac9aebf6b61f589eeb4"
format_stamp: "Formatted at 2026-09-26 09:29:49 on dist-test-slave-dfzm"
server_key: "ec03fd87db55b49c9ad5cbdb1b0e4609"
server_key_iv: "e269b6c555443e37f2d038b109db5c41"
server_key_version: "encryptionkey@0"
I20260926 09:29:49.646591 13823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:29:49.649873 13816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } attempt: 2
I20260926 09:29:49.650322 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.650197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45727 (local address 127.0.0.1:41720)
0926 09:29:49.650256 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.650294 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 09:29:49.650614 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.650537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45727 (local address 127.0.0.1:41736)
0926 09:29:49.650577 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.650598 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 09:29:49.650696 13816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45727: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45727: connect: Connection refused (error 111)
I20260926 09:29:49.679560 13823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:29:49.681998 13823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:29:49.682118 13823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:29:49.685886 13823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45727
I20260926 09:29:49.685967 13890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45727 every 8 connection(s)
I20260926 09:29:49.686321 13823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-2/data/info.pb
I20260926 09:29:49.687183 13891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:29:49.688735 13891 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:29:49.689817 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 13823
I20260926 09:29:49.689954 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414839069479-392-0/minicluster-data/master-2/wal/instance
I20260926 09:29:49.690145 392 external_mini_cluster.cc:1428] Setting key c629d7adf17f9eb6b0ffe1f131246c23
Sep 26 09:29:49 dist-test-slave-dfzm krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 09:29:49 dist-test-slave-dfzm krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 09:29:49.698832 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.689474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56942 (local address 127.0.98.62:44405)
0926 09:29:49.689536 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:49.689538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:49.689777 (+ 239us) server_negotiation.cc:407] Connection header received
0926 09:29:49.690304 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:49.690305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:49.690319 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:49.690376 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:49.691159 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.691819 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.693005 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.693300 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.693372 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:49.695473 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:49.695475 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:49.695475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:49.695483 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:49.697078 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.697725 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.697726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.697727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.697788 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.698139 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.698140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.698140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.698288 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:49.698365 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:49.698619 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 09:29:49.698705 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 09:29:49.698839 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.689430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44405 (local address 127.0.0.1:56942)
0926 09:29:49.689610 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.689676 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:49.690172 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:49.690450 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:49.690455 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:49.690830 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:29:49.691071 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:49.691075 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.691894 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.691895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:49.692930 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:49.692932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.693380 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.693381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:49.693487 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:49.693994 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:29:49.694004 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:29:49.695385 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:29:49.697199 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:49.697222 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:49.697224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:49.697638 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:49.697875 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:49.697876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:49.697876 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:49.698055 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:49.698359 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:29:49.698363 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:29:49.698473 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 09:29:49.698547 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 09:29:49.698656 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260926 09:29:49.699451 13891 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:29:49.699692 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.690567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.0.98.60:45727)
0926 09:29:49.690720 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:49.690725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:49.690745 (+ 20us) server_negotiation.cc:407] Connection header received
0926 09:29:49.690821 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:49.690824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:49.690847 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:49.690917 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:49.691589 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.692691 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.693635 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.694037 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.694111 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:49.696378 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:49.696392 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:49.696392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:49.696398 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:49.698304 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.698675 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.698679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.698679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.698735 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.698937 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.698938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.698938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.699116 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:49.699269 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:49.699486 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 09:29:49.699579 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
Sep 26 09:29:49 dist-test-slave-dfzm krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 09:29:49 dist-test-slave-dfzm krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:29:49.706655 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.699690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36505 (local address 127.0.0.1:42786)
0926 09:29:49.699790 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.699802 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:49.699849 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:49.699962 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:49.699963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:49.700121 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:29:49.700175 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:49.700177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.701099 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.701100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:49.701550 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:49.701552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.702069 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.702070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:49.702139 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:49.702461 (+ 322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:29:49.702466 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:29:49.703800 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:29:49.705625 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:49.705626 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:49.705626 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:49.705800 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:49.706030 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:49.706031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:49.706031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:49.706095 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:49.706342 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:29:49.706343 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:29:49.706385 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 09:29:49.706474 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 09:29:49.706562 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 09:29:49.706655 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.699736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42786 (local address 127.0.98.61:36505)
0926 09:29:49.699782 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:49.699784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:49.699838 (+ 54us) server_negotiation.cc:407] Connection header received
0926 09:29:49.699899 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:49.699900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:49.699909 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:49.699962 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:49.700259 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.701028 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.701683 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.701994 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.702054 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:49.703881 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:49.703881 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:49.703882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:49.703888 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:49.705489 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.705896 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.705897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.705898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.705960 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.706166 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.706166 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.706167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.706276 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:49.706350 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:49.706497 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 09:29:49.706568 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:29:49.706930 13741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } attempt: 2
I20260926 09:29:49.707093 13891 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:29:49.709905 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.701132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56950 (local address 127.0.98.62:44405)
0926 09:29:49.701191 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:49.701193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:49.701239 (+ 46us) server_negotiation.cc:407] Connection header received
0926 09:29:49.701315 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:49.701316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:49.701325 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:49.701368 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:49.701939 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.702578 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.703457 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.703689 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.703744 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:49.705965 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:49.705967 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:49.705967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:49.705974 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:49.707964 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.708306 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.708307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.708308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.708365 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.708557 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.708557 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.708558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.708685 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:49.709276 (+ 591us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:49.709715 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 09:29:49.709809 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 09:29:49 dist-test-slave-dfzm krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 09:29:49 dist-test-slave-dfzm krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 09:29:49 dist-test-slave-dfzm krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 09:29:49.715588 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.707270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45727 (local address 127.0.0.1:41762)
0926 09:29:49.707449 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.707458 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:49.707484 (+ 26us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:49.707742 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:49.707743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:49.707867 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:29:49.707891 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:49.707892 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.708755 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.708756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:49.709206 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:49.709207 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.709646 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.709647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:49.709695 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:49.710104 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:29:49.710109 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:29:49.711921 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:29:49.714433 (+ 2512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:49.714434 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:49.714434 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:49.714656 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:49.714873 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:49.714874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:49.714875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:49.714940 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:49.715238 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:29:49.715239 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:29:49.715278 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 09:29:49.715358 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 09:29:49.715460 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260926 09:29:49.715690 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.707156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45727 (local address 127.0.0.1:41758)
0926 09:29:49.707200 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.707209 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:49.707243 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:49.707423 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:49.707424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:49.707665 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:29:49.707707 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:49.707709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.708489 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.708490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:49.709266 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:49.709269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.709797 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.709798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:49.709851 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:49.710284 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:29:49.710289 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:29:49.711994 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:29:49.714609 (+ 2615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:49.714611 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:49.714611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:49.714818 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:49.715041 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:49.715042 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:49.715042 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:49.715095 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:49.715400 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:29:49.715401 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:29:49.715446 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 09:29:49.715494 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 09:29:49.715580 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 09:29:49.715701 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.707315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41762 (local address 127.0.98.60:45727)
0926 09:29:49.707637 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:49.707639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:49.707644 (+ 5us) server_negotiation.cc:407] Connection header received
0926 09:29:49.707657 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:49.707657 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:49.707663 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:49.707740 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:49.707968 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.708683 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.709297 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.709518 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.709588 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:49.712027 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:49.712028 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:49.712028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:49.712034 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:49.714334 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.714730 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.714731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.714732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.714795 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.715009 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.715009 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.715010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.715176 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:49.715246 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:49.715415 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 09:29:49.715520 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":307,"thread_start_us":92,"threads_started":1}
I20260926 09:29:49.716501 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.707182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41758 (local address 127.0.98.60:45727)
0926 09:29:49.707315 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:49.707317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:49.707323 (+ 6us) server_negotiation.cc:407] Connection header received
0926 09:29:49.707339 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:49.707339 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:49.707347 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:49.707418 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:49.707814 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.708425 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.709358 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.709710 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.709970 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:49.712079 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:49.712080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:49.712080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:49.712087 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:49.714531 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.714915 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.714916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.714916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.714958 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.715205 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.715206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.715206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.715319 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:49.716319 (+ 1000us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:49.716358 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 09:29:49.716431 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":119}
I20260926 09:29:49.717527 13891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c3295d97eba4ac9aebf6b61f589eeb4: Bootstrap starting.
I20260926 09:29:49.717924 13741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa9295dd63cf4ba78255db4aee6151b0: Bootstrap starting.
I20260926 09:29:49.718344 13891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c3295d97eba4ac9aebf6b61f589eeb4: Neither blocks nor log segments found. Creating new log.
I20260926 09:29:49.718405 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.710734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42790 (local address 127.0.98.61:36505)
0926 09:29:49.710771 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:49.710773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:49.710781 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:29:49.710832 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:49.710833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:49.710843 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:49.710909 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:49.711314 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.711782 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.712606 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.712799 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.712872 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:49.715661 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:49.715662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:49.715662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:49.715668 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:49.717192 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.717559 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.717560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.717560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.717647 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.717838 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.717839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.717840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.717950 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:49.718133 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:49.718215 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 09:29:49.718292 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:29:49.718637 13741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa9295dd63cf4ba78255db4aee6151b0: Neither blocks nor log segments found. Creating new log.
I20260926 09:29:49.718648 13891 log.cc:824] T 00000000000000000000000000000000 P 4c3295d97eba4ac9aebf6b61f589eeb4: Log is configured to *not* fsync() on all Append() calls
I20260926 09:29:49.718966 13741 log.cc:824] T 00000000000000000000000000000000 P fa9295dd63cf4ba78255db4aee6151b0: Log is configured to *not* fsync() on all Append() calls
I20260926 09:29:49.719442 13891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c3295d97eba4ac9aebf6b61f589eeb4: No bootstrap required, opened a new log
I20260926 09:29:49.719910 13741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa9295dd63cf4ba78255db4aee6151b0: No bootstrap required, opened a new log
I20260926 09:29:49.720113 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:29:49.720973 13891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c3295d97eba4ac9aebf6b61f589eeb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9295dd63cf4ba78255db4aee6151b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } } peers { permanent_uuid: "9954aab5698441c399f3b16df9857a35" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } } peers { permanent_uuid: "4c3295d97eba4ac9aebf6b61f589eeb4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } }
I20260926 09:29:49.721117 13891 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c3295d97eba4ac9aebf6b61f589eeb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:29:49.721161 13891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c3295d97eba4ac9aebf6b61f589eeb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c3295d97eba4ac9aebf6b61f589eeb4, State: Initialized, Role: FOLLOWER
I20260926 09:29:49.721276 13891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c3295d97eba4ac9aebf6b61f589eeb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9295dd63cf4ba78255db4aee6151b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } } peers { permanent_uuid: "9954aab5698441c399f3b16df9857a35" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } } peers { permanent_uuid: "4c3295d97eba4ac9aebf6b61f589eeb4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } }
I20260926 09:29:49.721618 13891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c3295d97eba4ac9aebf6b61f589eeb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:29:49.721544 13903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c3295d97eba4ac9aebf6b61f589eeb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9295dd63cf4ba78255db4aee6151b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } } peers { permanent_uuid: "9954aab5698441c399f3b16df9857a35" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } } peers { permanent_uuid: "4c3295d97eba4ac9aebf6b61f589eeb4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } } }
I20260926 09:29:49.721683 13903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c3295d97eba4ac9aebf6b61f589eeb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:29:49.722148 13741 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa9295dd63cf4ba78255db4aee6151b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9295dd63cf4ba78255db4aee6151b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } } peers { permanent_uuid: "9954aab5698441c399f3b16df9857a35" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } } peers { permanent_uuid: "4c3295d97eba4ac9aebf6b61f589eeb4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } }
I20260926 09:29:49.722286 13741 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa9295dd63cf4ba78255db4aee6151b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:29:49.722333 13741 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa9295dd63cf4ba78255db4aee6151b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9295dd63cf4ba78255db4aee6151b0, State: Initialized, Role: FOLLOWER
I20260926 09:29:49.722460 13741 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa9295dd63cf4ba78255db4aee6151b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9295dd63cf4ba78255db4aee6151b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } } peers { permanent_uuid: "9954aab5698441c399f3b16df9857a35" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } } peers { permanent_uuid: "4c3295d97eba4ac9aebf6b61f589eeb4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } }
I20260926 09:29:49.722710 13912 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa9295dd63cf4ba78255db4aee6151b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9295dd63cf4ba78255db4aee6151b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } } peers { permanent_uuid: "9954aab5698441c399f3b16df9857a35" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } } peers { permanent_uuid: "4c3295d97eba4ac9aebf6b61f589eeb4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } } }
I20260926 09:29:49.722771 13741 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa9295dd63cf4ba78255db4aee6151b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:29:49.722844 13912 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa9295dd63cf4ba78255db4aee6151b0 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 09:29:49.723522 13918 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c3295d97eba4ac9aebf6b61f589eeb4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:29:49.723583 13918 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:29:49.723732 13823 master_runner.cc:186] Error getting master registration for 127.0.98.62:44405: OK, instance_id { permanent_uuid: "fa9295dd63cf4ba78255db4aee6151b0" instance_seqno: 1790414989362303 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 09:29:49.726028 13823 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 09:29:49.725251 13673 master_runner.cc:186] Error getting master registration for 127.0.98.61:36505: OK, instance_id { permanent_uuid: "9954aab5698441c399f3b16df9857a35" instance_seqno: 1790414989506644 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 09:29:49.728330 13673 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 09:29:49.728900 13930 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa9295dd63cf4ba78255db4aee6151b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:29:49.728945 13930 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:29:49.731789 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.722010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42798 (local address 127.0.98.61:36505)
0926 09:29:49.722044 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:49.722046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:49.723814 (+ 1768us) server_negotiation.cc:407] Connection header received
0926 09:29:49.723927 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:49.723928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:49.723939 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:49.723972 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:49.724796 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.725476 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.726499 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.726792 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.726867 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:49.728591 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:49.728592 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:49.728592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:49.728599 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:49.730526 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.730944 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.730945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.730946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.731000 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.731195 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.731196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.731196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.731320 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:49.731393 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:49.731604 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 09:29:49.731681 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 09:29:49.737290 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.722215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56964 (local address 127.0.98.62:44405)
0926 09:29:49.722265 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:49.722267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:49.726582 (+ 4315us) server_negotiation.cc:407] Connection header received
0926 09:29:49.726948 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:49.726949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:49.726959 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:49.726992 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:49.728284 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.728959 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.731584 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.731859 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.731903 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:49.733824 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:49.733826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:49.733826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:49.733833 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:49.735495 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.736151 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.736152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.736152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.736215 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.736674 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.736675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.736675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.736828 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:49.736925 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:49.737117 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 09:29:49.737201 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 09:29:49.738565 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.722514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41774 (local address 127.0.98.60:45727)
0926 09:29:49.722550 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:49.722552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:49.726677 (+ 4125us) server_negotiation.cc:407] Connection header received
0926 09:29:49.726832 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:49.726833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:49.726845 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:49.726881 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:49.729976 (+ 3095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.730652 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.732247 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.732543 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.732588 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:49.735604 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:49.735606 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:49.735606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:49.735613 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:49.737200 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.737655 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.737656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.737656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.737715 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.737958 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.737960 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.737960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.738094 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:49.738187 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:49.738392 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 09:29:49.738474 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:29:49.741824 13816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } attempt: 3
Sep 26 09:29:49 dist-test-slave-dfzm krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 09:29:49.749940 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.742112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45727 (local address 127.0.0.1:41778)
0926 09:29:49.742168 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:49.742178 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:49.742219 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:49.742374 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:49.742376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:49.742541 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:29:49.742588 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:49.742590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.743508 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.743509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:49.744323 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:49.744326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.744801 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.744802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:49.744869 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:49.745342 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:29:49.745348 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:29:49.746896 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:29:49.748736 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:49.748737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:49.748738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:49.748973 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:49.749195 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:49.749196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:49.749196 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:49.749280 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:49.749551 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:29:49.749552 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:29:49.749624 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 09:29:49.749690 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 09:29:49.749834 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 09:29:49.750288 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:49.742165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41778 (local address 127.0.98.60:45727)
0926 09:29:49.742204 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:49.742206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:49.742214 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:29:49.742279 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:49.742280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:49.742289 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:49.742351 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:49.742670 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.743424 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.744424 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:49.744711 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:49.744783 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:49.747012 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:49.747013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:49.747013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:49.747020 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:49.748643 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.749061 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.749062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.749063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.749115 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:49.749351 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:49.749352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:49.749352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:49.749473 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:49.749526 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:49.750070 (+ 544us) server_negotiation.cc:299] Negotiation successful
0926 09:29:49.750175 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:29:49.751848 13816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35: Bootstrap starting.
I20260926 09:29:49.752580 13816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35: Neither blocks nor log segments found. Creating new log.
I20260926 09:29:49.752894 13816 log.cc:824] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35: Log is configured to *not* fsync() on all Append() calls
I20260926 09:29:49.753845 13816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35: No bootstrap required, opened a new log
I20260926 09:29:49.755856 13816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9295dd63cf4ba78255db4aee6151b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } } peers { permanent_uuid: "9954aab5698441c399f3b16df9857a35" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } } peers { permanent_uuid: "4c3295d97eba4ac9aebf6b61f589eeb4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } }
I20260926 09:29:49.756008 13816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:29:49.756043 13816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9954aab5698441c399f3b16df9857a35, State: Initialized, Role: FOLLOWER
I20260926 09:29:49.756166 13816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9295dd63cf4ba78255db4aee6151b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } } peers { permanent_uuid: "9954aab5698441c399f3b16df9857a35" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } } peers { permanent_uuid: "4c3295d97eba4ac9aebf6b61f589eeb4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } }
I20260926 09:29:49.756484 13816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:29:49.757205 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9295dd63cf4ba78255db4aee6151b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } } peers { permanent_uuid: "9954aab5698441c399f3b16df9857a35" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } } peers { permanent_uuid: "4c3295d97eba4ac9aebf6b61f589eeb4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } } }
I20260926 09:29:49.757440 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 09:29:49.758455 13944 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:29:49.758554 13944 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:29:49.760540 13748 master_runner.cc:263] No leader master found from master 9954aab5698441c399f3b16df9857a35
I20260926 09:29:49.760591 13748 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 09:29:49.793857 13933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:29:49.793998 13933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9295dd63cf4ba78255db4aee6151b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } } peers { permanent_uuid: "9954aab5698441c399f3b16df9857a35" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } } peers { permanent_uuid: "4c3295d97eba4ac9aebf6b61f589eeb4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } }
I20260926 09:29:49.794255 13933 leader_election.cc:288] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa9295dd63cf4ba78255db4aee6151b0 (127.0.98.62:44405), 4c3295d97eba4ac9aebf6b61f589eeb4 (127.0.98.60:45727)
I20260926 09:29:49.794660 13715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9954aab5698441c399f3b16df9857a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9295dd63cf4ba78255db4aee6151b0" is_pre_election: true
I20260926 09:29:49.794840 13715 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fa9295dd63cf4ba78255db4aee6151b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9954aab5698441c399f3b16df9857a35 in term 0.
I20260926 09:29:49.795006 13865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9954aab5698441c399f3b16df9857a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c3295d97eba4ac9aebf6b61f589eeb4" is_pre_election: true
I20260926 09:29:49.795154 13865 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4c3295d97eba4ac9aebf6b61f589eeb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9954aab5698441c399f3b16df9857a35 in term 0.
I20260926 09:29:49.795413 13765 leader_election.cc:302] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9954aab5698441c399f3b16df9857a35, fa9295dd63cf4ba78255db4aee6151b0; no voters:
I20260926 09:29:49.795583 13933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:29:49.795637 13933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:29:49.795667 13933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:29:49.796671 13933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9295dd63cf4ba78255db4aee6151b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } } peers { permanent_uuid: "9954aab5698441c399f3b16df9857a35" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } } peers { permanent_uuid: "4c3295d97eba4ac9aebf6b61f589eeb4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } }
I20260926 09:29:49.796805 13933 leader_election.cc:288] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [CANDIDATE]: Term 1 election: Requested vote from peers fa9295dd63cf4ba78255db4aee6151b0 (127.0.98.62:44405), 4c3295d97eba4ac9aebf6b61f589eeb4 (127.0.98.60:45727)
I20260926 09:29:49.797091 13865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9954aab5698441c399f3b16df9857a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c3295d97eba4ac9aebf6b61f589eeb4"
I20260926 09:29:49.797205 13865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c3295d97eba4ac9aebf6b61f589eeb4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:29:49.797652 13715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9954aab5698441c399f3b16df9857a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9295dd63cf4ba78255db4aee6151b0"
I20260926 09:29:49.797749 13715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa9295dd63cf4ba78255db4aee6151b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:29:49.798697 13715 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fa9295dd63cf4ba78255db4aee6151b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9954aab5698441c399f3b16df9857a35 in term 1.
I20260926 09:29:49.798902 13765 leader_election.cc:302] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9954aab5698441c399f3b16df9857a35, fa9295dd63cf4ba78255db4aee6151b0; no voters:
I20260926 09:29:49.799463 13865 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4c3295d97eba4ac9aebf6b61f589eeb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9954aab5698441c399f3b16df9857a35 in term 1.
I20260926 09:29:49.799675 13933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:29:49.799862 13933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [term 1 LEADER]: Becoming Leader. State: Replica: 9954aab5698441c399f3b16df9857a35, State: Running, Role: LEADER
I20260926 09:29:49.799969 13933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9295dd63cf4ba78255db4aee6151b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } } peers { permanent_uuid: "9954aab5698441c399f3b16df9857a35" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } } peers { permanent_uuid: "4c3295d97eba4ac9aebf6b61f589eeb4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } }
I20260926 09:29:49.800405 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9954aab5698441c399f3b16df9857a35. Latest consensus state: current_term: 1 leader_uuid: "9954aab5698441c399f3b16df9857a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9295dd63cf4ba78255db4aee6151b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } } peers { permanent_uuid: "9954aab5698441c399f3b16df9857a35" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } } peers { permanent_uuid: "4c3295d97eba4ac9aebf6b61f589eeb4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } } }
I20260926 09:29:49.800491 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [sys.catalog]: This master's current role is: LEADER
I20260926 09:29:49.800716 13948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:29:49.800981 13948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:29:49.803934 13865 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4c3295d97eba4ac9aebf6b61f589eeb4 [term 1 FOLLOWER]: Refusing update from remote peer 9954aab5698441c399f3b16df9857a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:29:49.804229 13933 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c3295d97eba4ac9aebf6b61f589eeb4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:29:49.805361 13903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c3295d97eba4ac9aebf6b61f589eeb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9954aab5698441c399f3b16df9857a35. Latest consensus state: current_term: 1 leader_uuid: "9954aab5698441c399f3b16df9857a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9295dd63cf4ba78255db4aee6151b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } } peers { permanent_uuid: "9954aab5698441c399f3b16df9857a35" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } } peers { permanent_uuid: "4c3295d97eba4ac9aebf6b61f589eeb4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } } }
I20260926 09:29:49.805473 13903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c3295d97eba4ac9aebf6b61f589eeb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:29:49.805846 13715 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fa9295dd63cf4ba78255db4aee6151b0 [term 1 FOLLOWER]: Refusing update from remote peer 9954aab5698441c399f3b16df9857a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:29:49.806109 13933 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa9295dd63cf4ba78255db4aee6151b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:29:49.807205 13912 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa9295dd63cf4ba78255db4aee6151b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9954aab5698441c399f3b16df9857a35. Latest consensus state: current_term: 1 leader_uuid: "9954aab5698441c399f3b16df9857a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9295dd63cf4ba78255db4aee6151b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } } peers { permanent_uuid: "9954aab5698441c399f3b16df9857a35" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } } peers { permanent_uuid: "4c3295d97eba4ac9aebf6b61f589eeb4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } } }
I20260926 09:29:49.807298 13912 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa9295dd63cf4ba78255db4aee6151b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:29:49.808286 13949 mvcc.cc:205] Tried to move back new op lower bound from 7333539798234480640 to 7333539798221189120. Current Snapshot: MvccSnapshot[applied={T|T < 7333539798234480640}]
I20260926 09:29:49.808574 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9954aab5698441c399f3b16df9857a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9295dd63cf4ba78255db4aee6151b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } } peers { permanent_uuid: "9954aab5698441c399f3b16df9857a35" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } } peers { permanent_uuid: "4c3295d97eba4ac9aebf6b61f589eeb4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } } }
I20260926 09:29:49.808657 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [sys.catalog]: This master's current role is: LEADER
I20260926 09:29:49.809010 13912 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa9295dd63cf4ba78255db4aee6151b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9954aab5698441c399f3b16df9857a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9295dd63cf4ba78255db4aee6151b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } } peers { permanent_uuid: "9954aab5698441c399f3b16df9857a35" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } } peers { permanent_uuid: "4c3295d97eba4ac9aebf6b61f589eeb4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } } }
I20260926 09:29:49.809098 13912 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa9295dd63cf4ba78255db4aee6151b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:29:49.809648 13952 mvcc.cc:205] Tried to move back new op lower bound from 7333539798234480640 to 7333539798221189120. Current Snapshot: MvccSnapshot[applied={T|T < 7333539798234480640}]
I20260926 09:29:49.810509 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9954aab5698441c399f3b16df9857a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9295dd63cf4ba78255db4aee6151b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } } peers { permanent_uuid: "9954aab5698441c399f3b16df9857a35" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } } peers { permanent_uuid: "4c3295d97eba4ac9aebf6b61f589eeb4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } } }
I20260926 09:29:49.810592 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35 [sys.catalog]: This master's current role is: LEADER
I20260926 09:29:49.810917 13903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c3295d97eba4ac9aebf6b61f589eeb4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9954aab5698441c399f3b16df9857a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9295dd63cf4ba78255db4aee6151b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44405 } } peers { permanent_uuid: "9954aab5698441c399f3b16df9857a35" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36505 } } peers { permanent_uuid: "4c3295d97eba4ac9aebf6b61f589eeb4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45727 } } }
I20260926 09:29:49.811003 13903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c3295d97eba4ac9aebf6b61f589eeb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:29:49.811609 13950 mvcc.cc:205] Tried to move back new op lower bound from 7333539798234480640 to 7333539798221189120. Current Snapshot: MvccSnapshot[applied={T|T < 7333539798234480640}]
I20260926 09:29:49.812534 13948 catalog_manager.cc:1347] Generated new cluster ID: 41966bd2c6eb4dc3ab6bbc583afe2cc5
I20260926 09:29:49.812592 13948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:29:49.823243 13948 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:29:49.824039 13948 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:29:49.832749 13948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9954aab5698441c399f3b16df9857a35: Generated new TSK 0
I20260926 09:29:49.832958 13948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:29:50.724025 13918 catalog_manager.cc:1259] Loaded cluster ID: 41966bd2c6eb4dc3ab6bbc583afe2cc5
I20260926 09:29:50.724128 13918 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4c3295d97eba4ac9aebf6b61f589eeb4: loading cluster ID for follower catalog manager: success
I20260926 09:29:50.725082 13918 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4c3295d97eba4ac9aebf6b61f589eeb4: acquiring CA information for follower catalog manager: success
I20260926 09:29:50.725411 13918 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4c3295d97eba4ac9aebf6b61f589eeb4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 09:29:50.729321 13930 catalog_manager.cc:1259] Loaded cluster ID: 41966bd2c6eb4dc3ab6bbc583afe2cc5
I20260926 09:29:50.729377 13930 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fa9295dd63cf4ba78255db4aee6151b0: loading cluster ID for follower catalog manager: success
I20260926 09:29:50.730214 13930 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fa9295dd63cf4ba78255db4aee6151b0: acquiring CA information for follower catalog manager: success
I20260926 09:29:50.730530 13930 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fa9295dd63cf4ba78255db4aee6151b0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 09:29:50.896719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 13673
I20260926 09:29:50.902257 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 13748
I20260926 09:29:50.907088 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 13823
2026-09-26T09:29:50Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2841 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:29:50 dist-test-slave-dfzm krb5kdc[13966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:29:50 dist-test-slave-dfzm krb5kdc[13966](info): set up 2 sockets
Sep 26 09:29:50 dist-test-slave-dfzm krb5kdc[13966](info): commencing operation
krb5kdc: starting...
W20260926 09:29:51.950029 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:29:52 dist-test-slave-dfzm krb5kdc[13966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:29:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:29:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:29:52.064468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45655
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:29:52.146411 13979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:29:52.146685 13979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:29:52.146723 13979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:29:52.147886 13979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:29:52.147960 13979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:29:52.147993 13979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:29:52.148023 13979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:29:52.148056 13979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:29:52.149545 13979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:29:52.149811 13979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:29:52.149996 13979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:29:52.154744 13984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:52.154901 13979 server_base.cc:1034] running on GCE node
W20260926 09:29:52.154778 13985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:52.154739 13987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:52.155287 13979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:29:52.155550 13979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:29:52.156701 13979 hybrid_clock.cc:648] HybridClock initialized: now 1790414992156687 us; error 30 us; skew 500 ppm
Sep 26 09:29:52 dist-test-slave-dfzm krb5kdc[13966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:52.159291 13979 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:29:52.159783 13979 webserver.cc:460] Webserver started at http://127.0.98.62:43629/ using document root <none> and password file <none>
I20260926 09:29:52.160024 13979 fs_manager.cc:359] Metadata directory not provided
I20260926 09:29:52.160094 13979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:29:52.160249 13979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:29:52.161309 13979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "a681c16988da4764b75aed9b16931ef4"
format_stamp: "Formatted at 2026-09-26 09:29:52 on dist-test-slave-dfzm"
server_key: "a660d9426438a6f8dd97ac4cd575d3df"
server_key_iv: "31f95c024c7fb541d141e9407f6799da"
server_key_version: "encryptionkey@0"
I20260926 09:29:52.161749 13979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "a681c16988da4764b75aed9b16931ef4"
format_stamp: "Formatted at 2026-09-26 09:29:52 on dist-test-slave-dfzm"
server_key: "a660d9426438a6f8dd97ac4cd575d3df"
server_key_iv: "31f95c024c7fb541d141e9407f6799da"
server_key_version: "encryptionkey@0"
I20260926 09:29:52.163081 13979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:29:52.163843 13994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:52.164038 13979 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:29:52.164142 13979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "a681c16988da4764b75aed9b16931ef4"
format_stamp: "Formatted at 2026-09-26 09:29:52 on dist-test-slave-dfzm"
server_key: "a660d9426438a6f8dd97ac4cd575d3df"
server_key_iv: "31f95c024c7fb541d141e9407f6799da"
server_key_version: "encryptionkey@0"
I20260926 09:29:52.164232 13979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:29:52.177695 13979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:29:52.180243 13979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:29:52.180426 13979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:29:52.184587 13979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45655
I20260926 09:29:52.184628 14046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45655 every 8 connection(s)
I20260926 09:29:52.184990 13979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:29:52.185907 14047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:29:52.188153 14047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a681c16988da4764b75aed9b16931ef4: Bootstrap starting.
I20260926 09:29:52.188678 14047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a681c16988da4764b75aed9b16931ef4: Neither blocks nor log segments found. Creating new log.
I20260926 09:29:52.188915 14047 log.cc:824] T 00000000000000000000000000000000 P a681c16988da4764b75aed9b16931ef4: Log is configured to *not* fsync() on all Append() calls
I20260926 09:29:52.189201 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 13979
I20260926 09:29:52.189311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:29:52.189517 392 external_mini_cluster.cc:1428] Setting key 8c4af3684e128cd2f7bd8666ff5ff9f5
I20260926 09:29:52.189839 14047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a681c16988da4764b75aed9b16931ef4: No bootstrap required, opened a new log
I20260926 09:29:52.191977 14047 raft_consensus.cc:348] T 00000000000000000000000000000000 P a681c16988da4764b75aed9b16931ef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a681c16988da4764b75aed9b16931ef4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45655 } }
I20260926 09:29:52.192160 14047 raft_consensus.cc:374] T 00000000000000000000000000000000 P a681c16988da4764b75aed9b16931ef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:29:52.192200 14047 raft_consensus.cc:729] T 00000000000000000000000000000000 P a681c16988da4764b75aed9b16931ef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a681c16988da4764b75aed9b16931ef4, State: Initialized, Role: FOLLOWER
I20260926 09:29:52.192353 14047 consensus_queue.cc:260] T 00000000000000000000000000000000 P a681c16988da4764b75aed9b16931ef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a681c16988da4764b75aed9b16931ef4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45655 } }
I20260926 09:29:52.192445 14047 raft_consensus.cc:388] T 00000000000000000000000000000000 P a681c16988da4764b75aed9b16931ef4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:29:52.192477 14047 raft_consensus.cc:482] T 00000000000000000000000000000000 P a681c16988da4764b75aed9b16931ef4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:29:52.192531 14047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a681c16988da4764b75aed9b16931ef4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:29:52.193246 14047 raft_consensus.cc:504] T 00000000000000000000000000000000 P a681c16988da4764b75aed9b16931ef4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a681c16988da4764b75aed9b16931ef4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45655 } }
I20260926 09:29:52.193360 14047 leader_election.cc:302] T 00000000000000000000000000000000 P a681c16988da4764b75aed9b16931ef4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a681c16988da4764b75aed9b16931ef4; no voters:
I20260926 09:29:52.193506 14047 leader_election.cc:288] T 00000000000000000000000000000000 P a681c16988da4764b75aed9b16931ef4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:29:52.193570 14052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a681c16988da4764b75aed9b16931ef4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:29:52.193784 14052 raft_consensus.cc:686] T 00000000000000000000000000000000 P a681c16988da4764b75aed9b16931ef4 [term 1 LEADER]: Becoming Leader. State: Replica: a681c16988da4764b75aed9b16931ef4, State: Running, Role: LEADER
I20260926 09:29:52.193946 14052 consensus_queue.cc:237] T 00000000000000000000000000000000 P a681c16988da4764b75aed9b16931ef4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a681c16988da4764b75aed9b16931ef4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45655 } }
I20260926 09:29:52.193768 14047 sys_catalog.cc:564] T 00000000000000000000000000000000 P a681c16988da4764b75aed9b16931ef4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:29:52.194339 14053 sys_catalog.cc:455] T 00000000000000000000000000000000 P a681c16988da4764b75aed9b16931ef4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a681c16988da4764b75aed9b16931ef4. Latest consensus state: current_term: 1 leader_uuid: "a681c16988da4764b75aed9b16931ef4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a681c16988da4764b75aed9b16931ef4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45655 } } }
I20260926 09:29:52.194562 14053 sys_catalog.cc:458] T 00000000000000000000000000000000 P a681c16988da4764b75aed9b16931ef4 [sys.catalog]: This master's current role is: LEADER
I20260926 09:29:52.194499 14054 sys_catalog.cc:455] T 00000000000000000000000000000000 P a681c16988da4764b75aed9b16931ef4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a681c16988da4764b75aed9b16931ef4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a681c16988da4764b75aed9b16931ef4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45655 } } }
I20260926 09:29:52.194826 14054 sys_catalog.cc:458] T 00000000000000000000000000000000 P a681c16988da4764b75aed9b16931ef4 [sys.catalog]: This master's current role is: LEADER
I20260926 09:29:52.195003 14056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:29:52.195462 14056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:29:52.197500 14056 catalog_manager.cc:1347] Generated new cluster ID: 70be24cf6ec14627a76dee510fd0b2f8
I20260926 09:29:52.197553 14056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:29:52 dist-test-slave-dfzm krb5kdc[13966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:29:52.202252 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:52.190067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32832 (local address 127.0.98.62:45655)
0926 09:29:52.190290 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:52.190300 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:52.190336 (+ 36us) server_negotiation.cc:407] Connection header received
0926 09:29:52.191167 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:52.191176 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:52.191242 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:52.191399 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:52.192004 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:52.193772 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:52.194851 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:52.195486 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:52.195941 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:52.198510 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:52.198514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:52.198514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:52.198537 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:52.200444 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:52.200820 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:52.200823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:52.200823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:52.200911 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:52.201070 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:52.201071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:52.201071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:52.201362 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:52.201497 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:52.201934 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 09:29:52.202058 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":119,"threads_started":1}
I20260926 09:29:52.203236 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:29:52.209323 14056 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:29:52.209935 14056 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:29:52.215380 14056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a681c16988da4764b75aed9b16931ef4: Generated new TSK 0
I20260926 09:29:52.215574 14056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:29:52.257109 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 13979
2026-09-26T09:29:52Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1348 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:29:52 dist-test-slave-dfzm krb5kdc[14074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:29:52 dist-test-slave-dfzm krb5kdc[14074](info): set up 2 sockets
Sep 26 09:29:52 dist-test-slave-dfzm krb5kdc[14074](info): commencing operation
krb5kdc: starting...
W20260926 09:29:53.349763 392 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:29:53 dist-test-slave-dfzm krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:29:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:29:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:29:53.468626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43703
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:29:53.550884 14087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:29:53.551162 14087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:29:53.551203 14087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:29:53.552371 14087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:29:53.552448 14087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:29:53.552479 14087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:29:53.552515 14087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:29:53.552541 14087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:29:53.554148 14087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:29:53.554452 14087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:29:53.554693 14087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:29:53.559648 14093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:53.559672 14092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:53.559672 14095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:53.559966 14087 server_base.cc:1034] running on GCE node
I20260926 09:29:53.560248 14087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:29:53.560591 14087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:29:53.561759 14087 hybrid_clock.cc:648] HybridClock initialized: now 1790414993561745 us; error 45 us; skew 500 ppm
Sep 26 09:29:53 dist-test-slave-dfzm krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:53.564378 14087 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:29:53.564864 14087 webserver.cc:460] Webserver started at http://127.0.98.62:40341/ using document root <none> and password file <none>
I20260926 09:29:53.565096 14087 fs_manager.cc:359] Metadata directory not provided
I20260926 09:29:53.565165 14087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:29:53.565317 14087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:29:53.566475 14087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "40ae0427ed804ab3b3305036c4ea228e"
format_stamp: "Formatted at 2026-09-26 09:29:53 on dist-test-slave-dfzm"
server_key: "12a495eabcfd8026e46757e694d525cd"
server_key_iv: "ce9a32e3d5f1f745b801bb13284c2a51"
server_key_version: "encryptionkey@0"
I20260926 09:29:53.566854 14087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "40ae0427ed804ab3b3305036c4ea228e"
format_stamp: "Formatted at 2026-09-26 09:29:53 on dist-test-slave-dfzm"
server_key: "12a495eabcfd8026e46757e694d525cd"
server_key_iv: "ce9a32e3d5f1f745b801bb13284c2a51"
server_key_version: "encryptionkey@0"
I20260926 09:29:53.568202 14087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:29:53.569043 14102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:53.569228 14087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:29:53.569334 14087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "40ae0427ed804ab3b3305036c4ea228e"
format_stamp: "Formatted at 2026-09-26 09:29:53 on dist-test-slave-dfzm"
server_key: "12a495eabcfd8026e46757e694d525cd"
server_key_iv: "ce9a32e3d5f1f745b801bb13284c2a51"
server_key_version: "encryptionkey@0"
I20260926 09:29:53.569410 14087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:29:53.584483 14087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:29:53.586638 14087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:29:53.586807 14087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:29:53.591322 14087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43703
I20260926 09:29:53.591351 14154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43703 every 8 connection(s)
I20260926 09:29:53.591754 14087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:29:53.592852 14155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:29:53.593359 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 14087
I20260926 09:29:53.593487 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:29:53.593691 392 external_mini_cluster.cc:1428] Setting key 388ebfc096d7aa0cce4d7dccbeff0fe7
I20260926 09:29:53.596076 14155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e: Bootstrap starting.
I20260926 09:29:53.596799 14155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e: Neither blocks nor log segments found. Creating new log.
I20260926 09:29:53.597126 14155 log.cc:824] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e: Log is configured to *not* fsync() on all Append() calls
I20260926 09:29:53.597961 14155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e: No bootstrap required, opened a new log
I20260926 09:29:53.600165 14155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ae0427ed804ab3b3305036c4ea228e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43703 } }
I20260926 09:29:53.600338 14155 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:29:53.600405 14155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40ae0427ed804ab3b3305036c4ea228e, State: Initialized, Role: FOLLOWER
I20260926 09:29:53.600570 14155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ae0427ed804ab3b3305036c4ea228e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43703 } }
I20260926 09:29:53.600665 14155 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:29:53.600733 14155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:29:53.600786 14155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:29:53.601464 14155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ae0427ed804ab3b3305036c4ea228e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43703 } }
I20260926 09:29:53.601606 14155 leader_election.cc:302] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40ae0427ed804ab3b3305036c4ea228e; no voters:
Sep 26 09:29:53 dist-test-slave-dfzm krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:29:53.601779 14155 leader_election.cc:288] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:29:53.601835 14160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:29:53.602036 14155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:29:53.602026 14160 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 1 LEADER]: Becoming Leader. State: Replica: 40ae0427ed804ab3b3305036c4ea228e, State: Running, Role: LEADER
I20260926 09:29:53.602258 14160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ae0427ed804ab3b3305036c4ea228e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43703 } }
I20260926 09:29:53.602725 14162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40ae0427ed804ab3b3305036c4ea228e. Latest consensus state: current_term: 1 leader_uuid: "40ae0427ed804ab3b3305036c4ea228e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ae0427ed804ab3b3305036c4ea228e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43703 } } }
I20260926 09:29:53.602877 14162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [sys.catalog]: This master's current role is: LEADER
I20260926 09:29:53.602715 14161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40ae0427ed804ab3b3305036c4ea228e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ae0427ed804ab3b3305036c4ea228e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43703 } } }
I20260926 09:29:53.603178 14161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [sys.catalog]: This master's current role is: LEADER
I20260926 09:29:53.603340 14169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:29:53.604130 14169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:29:53.605860 14169 catalog_manager.cc:1347] Generated new cluster ID: 4d2a152c303a4b6b976d22dabaffdda6
I20260926 09:29:53.605912 14169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:29:53.607008 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:53.594466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38574 (local address 127.0.98.62:43703)
0926 09:29:53.594678 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:53.594688 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:53.594723 (+ 35us) server_negotiation.cc:407] Connection header received
0926 09:29:53.595252 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:53.595260 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:53.595333 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:53.595488 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:53.596297 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:53.597680 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:53.598544 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:53.599181 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:53.599297 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:53.602391 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:53.602395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:53.602396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:53.602408 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:53.604835 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:53.605265 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:53.605268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:53.605269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:53.605375 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:53.605677 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:53.605678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:53.605679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:53.606004 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:53.606552 (+ 548us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:53.606635 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 09:29:53.606791 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":98,"threads_started":1}
I20260926 09:29:53.626730 14169 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:29:53.627310 14169 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:29:53.632154 14169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e: Generated new TSK 0
I20260926 09:29:53.632362 14169 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:29:53.660642 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43703
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:29:53.747888 14181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:29:53.748095 14181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:29:53.748119 14181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:29:53.749274 14181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:29:53.749316 14181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:29:53.749342 14181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:29:53.750856 14181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:29:53.751114 14181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:29:53.751354 14181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:29:53.756517 14189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:53.756731 14181 server_base.cc:1034] running on GCE node
W20260926 09:29:53.756527 14187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:53.756525 14186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:53.757025 14181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:29:53.757306 14181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:29:53.758471 14181 hybrid_clock.cc:648] HybridClock initialized: now 1790414993758438 us; error 49 us; skew 500 ppm
Sep 26 09:29:53 dist-test-slave-dfzm krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:53.761082 14181 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:29:53.761541 14181 webserver.cc:460] Webserver started at http://127.0.98.1:44303/ using document root <none> and password file <none>
I20260926 09:29:53.761802 14181 fs_manager.cc:359] Metadata directory not provided
I20260926 09:29:53.761852 14181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:29:53.761942 14181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:29:53.762931 14181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "25ee59e37ca2405aa4cc45582ed7ff30"
format_stamp: "Formatted at 2026-09-26 09:29:53 on dist-test-slave-dfzm"
server_key: "d2c0529cde0e230b2c2af1a712a72e07"
server_key_iv: "75998b18b744fb52e44f524e8445d697"
server_key_version: "encryptionkey@0"
I20260926 09:29:53.763226 14181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "25ee59e37ca2405aa4cc45582ed7ff30"
format_stamp: "Formatted at 2026-09-26 09:29:53 on dist-test-slave-dfzm"
server_key: "d2c0529cde0e230b2c2af1a712a72e07"
server_key_iv: "75998b18b744fb52e44f524e8445d697"
server_key_version: "encryptionkey@0"
I20260926 09:29:53.764499 14181 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:29:53.765154 14196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:53.765370 14181 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:29:53.765455 14181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "25ee59e37ca2405aa4cc45582ed7ff30"
format_stamp: "Formatted at 2026-09-26 09:29:53 on dist-test-slave-dfzm"
server_key: "d2c0529cde0e230b2c2af1a712a72e07"
server_key_iv: "75998b18b744fb52e44f524e8445d697"
server_key_version: "encryptionkey@0"
I20260926 09:29:53.765507 14181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:29:53.775621 14181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:29:53.777716 14181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:29:53.777835 14181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:29:53.778025 14181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:29:53.778347 14181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:29:53.778383 14181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:53.778438 14181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:29:53.778466 14181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:53.784256 14181 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34083
I20260926 09:29:53.784353 14309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34083 every 8 connection(s)
I20260926 09:29:53.784669 14181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:29:53.785787 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 14181
I20260926 09:29:53.785897 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:29:53.786108 392 external_mini_cluster.cc:1428] Setting key f8ea78b6f42409210600db8d388d042d
Sep 26 09:29:53 dist-test-slave-dfzm krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 09:29:53.796203 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:53.785429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46771 (local address 127.0.98.62:43703)
0926 09:29:53.785501 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:53.785503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:53.785823 (+ 320us) server_negotiation.cc:407] Connection header received
0926 09:29:53.786504 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:53.786504 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:53.786513 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:53.786550 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:53.787914 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:53.788461 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:53.790061 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:53.790288 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:53.790356 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:53.792931 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:53.792933 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:53.792933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:53.792941 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:53.794662 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:53.795184 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:53.795186 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:53.795186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:53.795242 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:53.795514 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:53.795515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:53.795515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:53.795637 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:53.795711 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:53.795993 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 09:29:53.796088 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 09:29:53.796407 14312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:53.785426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.1:46771)
0926 09:29:53.785654 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:53.785704 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:53.786389 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:53.786635 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:53.786639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:53.787502 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:29:53.787797 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:53.787803 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:53.788542 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:53.788543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:53.789929 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:53.789932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:53.790387 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:53.790388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:53.790539 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:53.791118 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:29:53.791129 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:29:53.792835 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:29:53.794758 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:53.794768 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:53.794769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:53.795002 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:53.795324 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:53.795325 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:53.795325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:53.795442 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:53.795714 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:29:53.795722 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:29:53.795855 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 09:29:53.795970 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 09:29:53.796131 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":90,"threads_started":1}
I20260926 09:29:53.796942 14310 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43703
I20260926 09:29:53.797062 14310 heartbeater.cc:464] Registering TS with master...
I20260926 09:29:53.797277 14310 heartbeater.cc:511] Master 127.0.98.62:43703 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 09:29:53.797919 14119 ts_manager.cc:194] Registered new tserver with Master: 25ee59e37ca2405aa4cc45582ed7ff30 (127.0.98.1:34083)
I20260926 09:29:53.798545 14119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:29:53.812630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43703
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:29:53.893679 14315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:29:53.893921 14315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:29:53.893959 14315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:29:53.895112 14315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:29:53.895180 14315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:29:53.895226 14315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:29:53.896736 14315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:29:53.897044 14315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:29:53.897285 14315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:29:53.902457 14315 server_base.cc:1034] running on GCE node
W20260926 09:29:53.902403 14321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:53.902448 14323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:53.902403 14320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:53.902937 14315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:29:53.903158 14315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:29:53.904304 14315 hybrid_clock.cc:648] HybridClock initialized: now 1790414993904285 us; error 33 us; skew 500 ppm
Sep 26 09:29:53 dist-test-slave-dfzm krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:53.906834 14315 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:29:53.907285 14315 webserver.cc:460] Webserver started at http://127.0.98.2:39879/ using document root <none> and password file <none>
I20260926 09:29:53.907474 14315 fs_manager.cc:359] Metadata directory not provided
I20260926 09:29:53.907532 14315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:29:53.907677 14315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:29:53.908721 14315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "64f187f1445a4645a8392ab36ed70c00"
format_stamp: "Formatted at 2026-09-26 09:29:53 on dist-test-slave-dfzm"
server_key: "36cbdeed0b5d2ac497e411407b6c2a14"
server_key_iv: "4e4af5dd6f93dab72bcefd5052492bca"
server_key_version: "encryptionkey@0"
I20260926 09:29:53.909075 14315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "64f187f1445a4645a8392ab36ed70c00"
format_stamp: "Formatted at 2026-09-26 09:29:53 on dist-test-slave-dfzm"
server_key: "36cbdeed0b5d2ac497e411407b6c2a14"
server_key_iv: "4e4af5dd6f93dab72bcefd5052492bca"
server_key_version: "encryptionkey@0"
I20260926 09:29:53.910482 14315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:29:53.911190 14330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:53.911401 14315 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:29:53.911475 14315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "64f187f1445a4645a8392ab36ed70c00"
format_stamp: "Formatted at 2026-09-26 09:29:53 on dist-test-slave-dfzm"
server_key: "36cbdeed0b5d2ac497e411407b6c2a14"
server_key_iv: "4e4af5dd6f93dab72bcefd5052492bca"
server_key_version: "encryptionkey@0"
I20260926 09:29:53.911561 14315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:29:53.919610 14315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:29:53.921799 14315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:29:53.921978 14315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:29:53.922237 14315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:29:53.922583 14315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:29:53.922623 14315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:53.922650 14315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:29:53.922712 14315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:53.929471 14315 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34545
I20260926 09:29:53.929507 14443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34545 every 8 connection(s)
I20260926 09:29:53.929908 14315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:29:53 dist-test-slave-dfzm krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:29:53.938057 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 14315
I20260926 09:29:53.938185 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:29:53.938366 392 external_mini_cluster.cc:1428] Setting key 1ce1f4c7217700eebdce3b6a5146003e
I20260926 09:29:53.940191 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:53.930724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36611 (local address 127.0.98.62:43703)
0926 09:29:53.930768 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:53.930770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:53.930951 (+ 181us) server_negotiation.cc:407] Connection header received
0926 09:29:53.931474 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:53.931476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:53.931489 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:53.931553 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:53.932485 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:53.932984 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:53.934041 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:53.934233 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:53.934308 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:53.936646 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:53.936648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:53.936648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:53.936657 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:53.938358 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:53.938904 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:53.938905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:53.938905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:53.938973 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:53.939391 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:53.939392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:53.939392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:53.939534 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:53.939623 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:53.939971 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 09:29:53.940078 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:29:53.940407 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:53.930681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.2:36611)
0926 09:29:53.930826 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:53.930868 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:53.931371 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:53.931621 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:53.931625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:53.932199 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:29:53.932383 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:53.932387 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:53.933080 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:53.933081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:53.933944 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:53.933947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:53.934314 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:53.934316 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:53.934436 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:53.935077 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:29:53.935085 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:29:53.936543 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:29:53.938471 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:53.938479 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:53.938481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:53.938801 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:53.939125 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:53.939126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:53.939126 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:53.939289 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:53.939625 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:29:53.939633 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:29:53.939789 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 09:29:53.939928 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 09:29:53.940113 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
I20260926 09:29:53.940987 14444 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43703
I20260926 09:29:53.941111 14444 heartbeater.cc:464] Registering TS with master...
I20260926 09:29:53.941321 14444 heartbeater.cc:511] Master 127.0.98.62:43703 requested a full tablet report, sending...
I20260926 09:29:53.941798 14119 ts_manager.cc:194] Registered new tserver with Master: 64f187f1445a4645a8392ab36ed70c00 (127.0.98.2:34545)
I20260926 09:29:53.942276 14119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36611
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:29:53.964309 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43703
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:29:54.045768 14449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:29:54.045982 14449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:29:54.046007 14449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:29:54.047122 14449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:29:54.047159 14449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:29:54.047184 14449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:29:54.048596 14449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:29:54.048815 14449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:29:54.048987 14449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:29:54.054092 14454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:54.054092 14457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:54.054163 14455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:54.054361 14449 server_base.cc:1034] running on GCE node
I20260926 09:29:54.054566 14449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:29:54.054865 14449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:29:54.056042 14449 hybrid_clock.cc:648] HybridClock initialized: now 1790414994056021 us; error 35 us; skew 500 ppm
Sep 26 09:29:54 dist-test-slave-dfzm krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:54.058676 14449 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:29:54.059257 14449 webserver.cc:460] Webserver started at http://127.0.98.3:35389/ using document root <none> and password file <none>
I20260926 09:29:54.059489 14449 fs_manager.cc:359] Metadata directory not provided
I20260926 09:29:54.059533 14449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:29:54.059674 14449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:29:54.060756 14449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "4ae513242c614266911e5618e941962f"
format_stamp: "Formatted at 2026-09-26 09:29:54 on dist-test-slave-dfzm"
server_key: "089958be2f78c6b866ff185eb65ce328"
server_key_iv: "3c440b33972e576d7b8626cab3e70954"
server_key_version: "encryptionkey@0"
I20260926 09:29:54.061120 14449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4ae513242c614266911e5618e941962f"
format_stamp: "Formatted at 2026-09-26 09:29:54 on dist-test-slave-dfzm"
server_key: "089958be2f78c6b866ff185eb65ce328"
server_key_iv: "3c440b33972e576d7b8626cab3e70954"
server_key_version: "encryptionkey@0"
I20260926 09:29:54.062497 14449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:29:54.063189 14464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:54.063369 14449 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:54.063462 14449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "4ae513242c614266911e5618e941962f"
format_stamp: "Formatted at 2026-09-26 09:29:54 on dist-test-slave-dfzm"
server_key: "089958be2f78c6b866ff185eb65ce328"
server_key_iv: "3c440b33972e576d7b8626cab3e70954"
server_key_version: "encryptionkey@0"
I20260926 09:29:54.063534 14449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:29:54.090022 14449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:29:54.092033 14449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:29:54.092165 14449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:29:54.092360 14449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:29:54.092609 14449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:29:54.092639 14449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:54.092662 14449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:29:54.092674 14449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:54.098780 14449 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41347
I20260926 09:29:54.098829 14577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41347 every 8 connection(s)
I20260926 09:29:54.099251 14449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:29:54.099388 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 14449
I20260926 09:29:54.099524 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:29:54.099761 392 external_mini_cluster.cc:1428] Setting key 22b372940552ec924cd532749c76c902
Sep 26 09:29:54 dist-test-slave-dfzm krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:29:54.108660 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.100094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37183 (local address 127.0.98.62:43703)
0926 09:29:54.100150 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:54.100152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:54.100349 (+ 197us) server_negotiation.cc:407] Connection header received
0926 09:29:54.100833 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:54.100834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:54.100852 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:54.100901 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:54.101883 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.102450 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.103613 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.103805 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.103863 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.105930 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:54.105932 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:54.105932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:54.105938 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:54.107405 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:54.107765 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:54.107766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:54.107766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:54.107817 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:54.108048 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:54.108049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:54.108050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:54.108171 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:54.108246 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:54.108514 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 09:29:54.108580 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:29:54.108735 14580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.100066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.3:37183)
0926 09:29:54.100229 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:54.100273 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:54.100760 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:54.100955 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:54.100958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:54.101521 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:29:54.101808 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:54.101814 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.102528 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.102529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:54.103522 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:54.103525 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.103885 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.103886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:54.103989 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.104448 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:29:54.104457 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:29:54.105847 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:29:54.107481 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:54.107487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:54.107488 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:54.107699 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:54.107879 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:54.107879 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:54.107879 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:54.107971 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:54.108244 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:29:54.108248 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:29:54.108377 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 09:29:54.108463 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 09:29:54.108579 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260926 09:29:54.109220 14578 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43703
I20260926 09:29:54.109299 14578 heartbeater.cc:464] Registering TS with master...
I20260926 09:29:54.109524 14578 heartbeater.cc:511] Master 127.0.98.62:43703 requested a full tablet report, sending...
I20260926 09:29:54.109874 14119 ts_manager.cc:194] Registered new tserver with Master: 4ae513242c614266911e5618e941962f (127.0.98.3:41347)
I20260926 09:29:54.110244 14119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37183
I20260926 09:29:54.111449 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:29:54.118882 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.112645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38582 (local address 127.0.98.62:43703)
0926 09:29:54.112702 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:54.112704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:54.112854 (+ 150us) server_negotiation.cc:407] Connection header received
0926 09:29:54.112876 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:54.112877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:54.112886 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:54.112943 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:54.113454 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.113990 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.114796 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.114983 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.115055 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.116253 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:54.116254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:54.116255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:54.116262 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:54.117712 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:54.118071 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:54.118072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:54.118073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:54.118131 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:54.118326 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:54.118327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:54.118328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:54.118476 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:54.118587 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:54.118699 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 09:29:54.118780 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 09:29:54.120496 14119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:29:54.122077 14119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:29:54.128556 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.124051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34083 (local address 127.0.0.1:53210)
0926 09:29:54.124501 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:54.124542 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:54.124644 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:54.124943 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:54.124947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:54.124949 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:29:54.125029 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:54.125031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.126397 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.126398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:54.127568 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:54.127569 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.128224 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.128225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:54.128322 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.128335 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 09:29:54.128424 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 09:29:54.128479 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":124,"thread_start_us":71,"threads_started":1}
I20260926 09:29:54.128566 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.124162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53210 (local address 127.0.98.1:34083)
0926 09:29:54.124364 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:54.124368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:54.124640 (+ 272us) server_negotiation.cc:407] Connection header received
0926 09:29:54.124756 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:54.124759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:54.124776 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:54.124851 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:29:54.125128 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.126288 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.127645 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.128134 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.128225 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.128284 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:54.128452 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 09:29:54.128490 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":72,"threads_started":1}
I20260926 09:29:54.128736 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.124127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56778 (local address 127.0.98.2:34545)
0926 09:29:54.124382 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:54.124387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:54.124776 (+ 389us) server_negotiation.cc:407] Connection header received
0926 09:29:54.124829 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:54.124831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:54.124846 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:54.124913 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:29:54.125316 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.126400 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.127572 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.128250 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.128330 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.128376 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:54.128621 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 09:29:54.128661 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":84,"threads_started":1}
I20260926 09:29:54.128867 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.124346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50974 (local address 127.0.98.3:41347)
0926 09:29:54.124495 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:54.124500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:54.124984 (+ 484us) server_negotiation.cc:407] Connection header received
0926 09:29:54.125028 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:54.125030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:54.125046 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:54.125122 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:29:54.125276 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.126056 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.127525 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.128250 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.128320 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.128374 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:54.128724 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 09:29:54.128775 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":68,"threads_started":1}
I20260926 09:29:54.128660 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.124229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41347 (local address 127.0.0.1:50974)
0926 09:29:54.124501 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:54.124544 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:54.124644 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:54.125132 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:54.125133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:54.125135 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:29:54.125191 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:54.125193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.126154 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.126156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:54.127430 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:54.127433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.128412 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.128413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:54.128482 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.128485 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:29:54.128534 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 09:29:54.128580 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":4,"thread_start_us":50,"threads_started":1}
I20260926 09:29:54.128871 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.124029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34545 (local address 127.0.0.1:56778)
0926 09:29:54.124501 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:54.124544 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:54.124644 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:54.125194 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:54.125195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:54.125197 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:29:54.125251 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:54.125253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.126501 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.126502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:54.127452 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:54.127454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.128412 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.128413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:54.128478 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.128482 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:29:54.128534 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 09:29:54.128582 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":2,"thread_start_us":59,"threads_started":1}
I20260926 09:29:54.129714 14244 tablet_service.cc:1467] Processing CreateTablet for tablet f68984d78df942eebe911fc5d613d83c (DEFAULT_TABLE table=test-table [id=182d175752f94c21aa22240de455401c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:29:54.129828 14512 tablet_service.cc:1467] Processing CreateTablet for tablet f68984d78df942eebe911fc5d613d83c (DEFAULT_TABLE table=test-table [id=182d175752f94c21aa22240de455401c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:29:54.130070 14244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68984d78df942eebe911fc5d613d83c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:29:54.130174 14512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68984d78df942eebe911fc5d613d83c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:29:54.130173 14378 tablet_service.cc:1467] Processing CreateTablet for tablet f68984d78df942eebe911fc5d613d83c (DEFAULT_TABLE table=test-table [id=182d175752f94c21aa22240de455401c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:29:54.130461 14378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68984d78df942eebe911fc5d613d83c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:29:54.132941 14592 tablet_bootstrap.cc:492] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f: Bootstrap starting.
I20260926 09:29:54.133352 14593 tablet_bootstrap.cc:492] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30: Bootstrap starting.
I20260926 09:29:54.133561 14594 tablet_bootstrap.cc:492] T f68984d78df942eebe911fc5d613d83c P 64f187f1445a4645a8392ab36ed70c00: Bootstrap starting.
I20260926 09:29:54.133694 14592 tablet_bootstrap.cc:654] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f: Neither blocks nor log segments found. Creating new log.
I20260926 09:29:54.134115 14593 tablet_bootstrap.cc:654] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30: Neither blocks nor log segments found. Creating new log.
I20260926 09:29:54.134287 14594 tablet_bootstrap.cc:654] T f68984d78df942eebe911fc5d613d83c P 64f187f1445a4645a8392ab36ed70c00: Neither blocks nor log segments found. Creating new log.
I20260926 09:29:54.134326 14592 log.cc:824] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f: Log is configured to *not* fsync() on all Append() calls
I20260926 09:29:54.134685 14593 log.cc:824] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30: Log is configured to *not* fsync() on all Append() calls
I20260926 09:29:54.135416 14592 tablet_bootstrap.cc:492] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f: No bootstrap required, opened a new log
I20260926 09:29:54.135483 14592 ts_tablet_manager.cc:1397] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 09:29:54.135531 14593 tablet_bootstrap.cc:492] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30: No bootstrap required, opened a new log
I20260926 09:29:54.135587 14593 ts_tablet_manager.cc:1397] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:29:54.135879 14594 log.cc:824] T f68984d78df942eebe911fc5d613d83c P 64f187f1445a4645a8392ab36ed70c00: Log is configured to *not* fsync() on all Append() calls
I20260926 09:29:54.136754 14594 tablet_bootstrap.cc:492] T f68984d78df942eebe911fc5d613d83c P 64f187f1445a4645a8392ab36ed70c00: No bootstrap required, opened a new log
I20260926 09:29:54.136832 14594 ts_tablet_manager.cc:1397] T f68984d78df942eebe911fc5d613d83c P 64f187f1445a4645a8392ab36ed70c00: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 09:29:54.137897 14592 raft_consensus.cc:348] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } } peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } }
I20260926 09:29:54.138067 14592 raft_consensus.cc:374] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:29:54.138101 14592 raft_consensus.cc:729] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae513242c614266911e5618e941962f, State: Initialized, Role: FOLLOWER
I20260926 09:29:54.138285 14592 consensus_queue.cc:260] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } } peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } }
I20260926 09:29:54.138573 14592 ts_tablet_manager.cc:1428] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 09:29:54.138707 14578 heartbeater.cc:503] Master 127.0.98.62:43703 was elected leader, sending a full tablet report...
I20260926 09:29:54.139232 14594 raft_consensus.cc:348] T f68984d78df942eebe911fc5d613d83c P 64f187f1445a4645a8392ab36ed70c00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } } peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } }
I20260926 09:29:54.139417 14594 raft_consensus.cc:374] T f68984d78df942eebe911fc5d613d83c P 64f187f1445a4645a8392ab36ed70c00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:29:54.139501 14594 raft_consensus.cc:729] T f68984d78df942eebe911fc5d613d83c P 64f187f1445a4645a8392ab36ed70c00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f187f1445a4645a8392ab36ed70c00, State: Initialized, Role: FOLLOWER
I20260926 09:29:54.139662 14594 consensus_queue.cc:260] T f68984d78df942eebe911fc5d613d83c P 64f187f1445a4645a8392ab36ed70c00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } } peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } }
I20260926 09:29:54.139791 14593 raft_consensus.cc:348] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } } peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } }
I20260926 09:29:54.139977 14594 ts_tablet_manager.cc:1428] T f68984d78df942eebe911fc5d613d83c P 64f187f1445a4645a8392ab36ed70c00: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 09:29:54.140005 14593 raft_consensus.cc:374] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:29:54.140069 14593 raft_consensus.cc:729] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ee59e37ca2405aa4cc45582ed7ff30, State: Initialized, Role: FOLLOWER
I20260926 09:29:54.140174 14444 heartbeater.cc:503] Master 127.0.98.62:43703 was elected leader, sending a full tablet report...
I20260926 09:29:54.140187 14593 consensus_queue.cc:260] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } } peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } }
I20260926 09:29:54.140461 14593 ts_tablet_manager.cc:1428] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260926 09:29:54.140583 14310 heartbeater.cc:503] Master 127.0.98.62:43703 was elected leader, sending a full tablet report...
W20260926 09:29:54.180536 14445 tablet.cc:2369] T f68984d78df942eebe911fc5d613d83c P 64f187f1445a4645a8392ab36ed70c00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:29:54.250488 14598 raft_consensus.cc:482] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:29:54.250643 14598 raft_consensus.cc:504] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } } peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } }
I20260926 09:29:54.251005 14598 leader_election.cc:288] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64f187f1445a4645a8392ab36ed70c00 (127.0.98.2:34545), 25ee59e37ca2405aa4cc45582ed7ff30 (127.0.98.1:34083)
I20260926 09:29:54.255015 14580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.251207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34083 (local address 127.0.98.3:39287)
0926 09:29:54.251272 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:54.251285 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:54.251338 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:54.251521 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:54.251522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:54.251524 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:29:54.251600 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:54.251603 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.252558 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.252558 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:54.253956 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:54.253960 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.254724 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.254725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:54.254809 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.254811 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:29:54.254873 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 09:29:54.254916 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":82}
I20260926 09:29:54.255023 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.251334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39287 (local address 127.0.98.1:34083)
0926 09:29:54.251379 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:54.251380 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:54.251389 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:29:54.251418 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:54.251419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:54.251430 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:54.251491 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:29:54.251685 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.252452 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.254053 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.254631 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.254710 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.254721 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:54.254908 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 09:29:54.254950 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:29:54.255023 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.251299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52175 (local address 127.0.98.2:34545)
0926 09:29:54.251352 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:54.251354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:54.251577 (+ 223us) server_negotiation.cc:407] Connection header received
0926 09:29:54.251604 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:54.251605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:54.251617 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:54.251687 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:29:54.251831 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.252500 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.254053 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.254631 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.254710 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.254721 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:54.254909 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 09:29:54.254950 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:29:54.255021 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.251220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34545 (local address 127.0.98.3:52175)
0926 09:29:54.251496 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:54.251506 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:54.251555 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:54.251706 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:54.251707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:54.251708 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:29:54.251762 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:54.251763 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.252573 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.252573 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:54.253956 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:54.253960 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.254724 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.254725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:54.254806 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.254810 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:29:54.254887 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 09:29:54.254916 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":142,"thread_start_us":117,"threads_started":1}
I20260926 09:29:54.255527 14264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68984d78df942eebe911fc5d613d83c" candidate_uuid: "4ae513242c614266911e5618e941962f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" is_pre_election: true
I20260926 09:29:54.255551 14398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68984d78df942eebe911fc5d613d83c" candidate_uuid: "4ae513242c614266911e5618e941962f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f187f1445a4645a8392ab36ed70c00" is_pre_election: true
I20260926 09:29:54.255694 14398 raft_consensus.cc:2454] T f68984d78df942eebe911fc5d613d83c P 64f187f1445a4645a8392ab36ed70c00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ae513242c614266911e5618e941962f in term 0.
I20260926 09:29:54.255694 14264 raft_consensus.cc:2454] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ae513242c614266911e5618e941962f in term 0.
I20260926 09:29:54.255941 14467 leader_election.cc:302] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ae513242c614266911e5618e941962f, 64f187f1445a4645a8392ab36ed70c00; no voters:
I20260926 09:29:54.256062 14598 raft_consensus.cc:2781] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:29:54.256115 14598 raft_consensus.cc:482] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:29:54.256139 14598 raft_consensus.cc:3037] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:29:54.256835 14598 raft_consensus.cc:504] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } } peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } }
I20260926 09:29:54.256989 14598 leader_election.cc:288] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f [CANDIDATE]: Term 1 election: Requested vote from peers 64f187f1445a4645a8392ab36ed70c00 (127.0.98.2:34545), 25ee59e37ca2405aa4cc45582ed7ff30 (127.0.98.1:34083)
I20260926 09:29:54.257143 14398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68984d78df942eebe911fc5d613d83c" candidate_uuid: "4ae513242c614266911e5618e941962f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f187f1445a4645a8392ab36ed70c00"
I20260926 09:29:54.257185 14264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68984d78df942eebe911fc5d613d83c" candidate_uuid: "4ae513242c614266911e5618e941962f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25ee59e37ca2405aa4cc45582ed7ff30"
I20260926 09:29:54.257243 14398 raft_consensus.cc:3037] T f68984d78df942eebe911fc5d613d83c P 64f187f1445a4645a8392ab36ed70c00 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:29:54.257254 14264 raft_consensus.cc:3037] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:29:54.258201 14264 raft_consensus.cc:2454] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ae513242c614266911e5618e941962f in term 1.
I20260926 09:29:54.258200 14398 raft_consensus.cc:2454] T f68984d78df942eebe911fc5d613d83c P 64f187f1445a4645a8392ab36ed70c00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ae513242c614266911e5618e941962f in term 1.
I20260926 09:29:54.258384 14467 leader_election.cc:302] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ae513242c614266911e5618e941962f, 64f187f1445a4645a8392ab36ed70c00; no voters:
I20260926 09:29:54.258512 14598 raft_consensus.cc:2781] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:29:54.258708 14598 raft_consensus.cc:686] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f [term 1 LEADER]: Becoming Leader. State: Replica: 4ae513242c614266911e5618e941962f, State: Running, Role: LEADER
I20260926 09:29:54.258807 14598 consensus_queue.cc:237] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } } peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } }
I20260926 09:29:54.259676 14119 catalog_manager.cc:5495] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ae513242c614266911e5618e941962f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4ae513242c614266911e5618e941962f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } health_report { overall_health: HEALTHY } } }
I20260926 09:29:54.274878 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.272257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50978 (local address 127.0.98.3:41347)
0926 09:29:54.272306 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:54.272308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:54.272316 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:29:54.272409 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:54.272410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:54.272430 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:54.272516 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:29:54.272659 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.273207 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.274042 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.274254 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.274344 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.274515 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:29:54.274624 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:29:54.274720 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:54.274764 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 09:29:54.274786 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 09:29:54.276582 14264 raft_consensus.cc:1261] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Refusing update from remote peer 4ae513242c614266911e5618e941962f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:29:54.276582 14398 raft_consensus.cc:1261] T f68984d78df942eebe911fc5d613d83c P 64f187f1445a4645a8392ab36ed70c00 [term 1 FOLLOWER]: Refusing update from remote peer 4ae513242c614266911e5618e941962f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:29:54.276942 14602 consensus_queue.cc:1035] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f [LEADER]: Connected to new peer: Peer: permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:29:54.277091 14598 consensus_queue.cc:1035] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:29:54.279547 14605 mvcc.cc:205] Tried to move back new op lower bound from 7333539816554119168 to 7333539816484737024. Current Snapshot: MvccSnapshot[applied={T|T < 7333539816554119168}]
I20260926 09:29:54.279639 14606 mvcc.cc:205] Tried to move back new op lower bound from 7333539816554119168 to 7333539816484737024. Current Snapshot: MvccSnapshot[applied={T|T < 7333539816554119168}]
I20260926 09:29:54.280225 14607 mvcc.cc:205] Tried to move back new op lower bound from 7333539816554119168 to 7333539816484737024. Current Snapshot: MvccSnapshot[applied={T|T < 7333539816554119168}]
I20260926 09:29:54.281973 14119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:29:54.282064 14119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:29:54.282802 14119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e: Sending DeleteTablet for 3 replicas of tablet f68984d78df942eebe911fc5d613d83c
I20260926 09:29:54.283100 14378 tablet_service.cc:1514] Processing DeleteTablet for tablet f68984d78df942eebe911fc5d613d83c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:29:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56778
I20260926 09:29:54.283180 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 14181
I20260926 09:29:54.283174 14512 tablet_service.cc:1514] Processing DeleteTablet for tablet f68984d78df942eebe911fc5d613d83c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:29:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50974
I20260926 09:29:54.284083 14615 tablet_replica.cc:330] T f68984d78df942eebe911fc5d613d83c P 64f187f1445a4645a8392ab36ed70c00: stopping tablet replica
I20260926 09:29:54.284114 14616 tablet_replica.cc:330] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f: stopping tablet replica
I20260926 09:29:54.284225 14615 raft_consensus.cc:2229] T f68984d78df942eebe911fc5d613d83c P 64f187f1445a4645a8392ab36ed70c00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:29:54.284245 14616 raft_consensus.cc:2229] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:29:54.284337 14615 raft_consensus.cc:2258] T f68984d78df942eebe911fc5d613d83c P 64f187f1445a4645a8392ab36ed70c00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:29:54.284363 14616 raft_consensus.cc:2258] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:29:54.284796 14615 ts_tablet_manager.cc:1905] T f68984d78df942eebe911fc5d613d83c P 64f187f1445a4645a8392ab36ed70c00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:29:54.286422 14615 ts_tablet_manager.cc:1918] T f68984d78df942eebe911fc5d613d83c P 64f187f1445a4645a8392ab36ed70c00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:29:54.286500 14615 log.cc:1196] T f68984d78df942eebe911fc5d613d83c P 64f187f1445a4645a8392ab36ed70c00: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/wal/wals/f68984d78df942eebe911fc5d613d83c
I20260926 09:29:54.286762 14615 ts_tablet_manager.cc:1939] T f68984d78df942eebe911fc5d613d83c P 64f187f1445a4645a8392ab36ed70c00: Deleting consensus metadata
I20260926 09:29:54.287254 14105 catalog_manager.cc:4841] TS 64f187f1445a4645a8392ab36ed70c00 (127.0.98.2:34545): tablet f68984d78df942eebe911fc5d613d83c (table test-table [id=182d175752f94c21aa22240de455401c]) successfully deleted
W20260926 09:29:54.288373 14103 connection.cc:535] client connection to 127.0.98.1:34083 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 09:29:54.288506 14103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 09:29:54.288826 14616 ts_tablet_manager.cc:1905] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:29:54.289004 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.288872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34083 (local address 127.0.0.1:53218)
0926 09:29:54.288927 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:54.288963 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 09:29:54.289125 14103 catalog_manager.cc:4584] TS 25ee59e37ca2405aa4cc45582ed7ff30 (127.0.98.1:34083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f68984d78df942eebe911fc5d613d83c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34083: connect: Connection refused (error 111)
I20260926 09:29:54.289951 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 14315
I20260926 09:29:54.294050 14616 ts_tablet_manager.cc:1918] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:29:54.294139 14616 log.cc:1196] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/f68984d78df942eebe911fc5d613d83c
I20260926 09:29:54.294400 14616 ts_tablet_manager.cc:1939] T f68984d78df942eebe911fc5d613d83c P 4ae513242c614266911e5618e941962f: Deleting consensus metadata
I20260926 09:29:54.294946 14105 catalog_manager.cc:4841] TS 4ae513242c614266911e5618e941962f (127.0.98.3:41347): tablet f68984d78df942eebe911fc5d613d83c (table test-table [id=182d175752f94c21aa22240de455401c]) successfully deleted
I20260926 09:29:54.295130 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 14449
I20260926 09:29:54.300088 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 14087
I20260926 09:29:54.305361 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43703
--webserver_interface=127.0.98.62
--webserver_port=40341
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:29:54.388451 14618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:29:54.388664 14618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:29:54.388687 14618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:29:54.389940 14618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:29:54.389981 14618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:29:54.389997 14618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:29:54.390040 14618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:29:54.390058 14618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:29:54.391520 14618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:29:54.391783 14618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:29:54.392011 14618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:29:54.400642 14627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:54.400642 14625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:54.400734 14624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:54.400861 14618 server_base.cc:1034] running on GCE node
I20260926 09:29:54.401186 14618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:29:54.401721 14618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:29:54.402900 14618 hybrid_clock.cc:648] HybridClock initialized: now 1790414994402890 us; error 28 us; skew 500 ppm
Sep 26 09:29:54 dist-test-slave-dfzm krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:54.405445 14618 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:29:54.406008 14618 webserver.cc:460] Webserver started at http://127.0.98.62:40341/ using document root <none> and password file <none>
I20260926 09:29:54.406260 14618 fs_manager.cc:359] Metadata directory not provided
I20260926 09:29:54.406347 14618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:29:54.407761 14618 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:29:54.408629 14634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:54.408819 14618 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:29:54.408924 14618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "40ae0427ed804ab3b3305036c4ea228e"
format_stamp: "Formatted at 2026-09-26 09:29:53 on dist-test-slave-dfzm"
server_key: "12a495eabcfd8026e46757e694d525cd"
server_key_iv: "ce9a32e3d5f1f745b801bb13284c2a51"
server_key_version: "encryptionkey@0"
I20260926 09:29:54.409449 14618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:29:54.425237 14618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:29:54.427371 14618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:29:54.427526 14618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:29:54.431658 14618 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43703
I20260926 09:29:54.431694 14686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43703 every 8 connection(s)
I20260926 09:29:54.432089 14618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:29:54.433773 14687 sys_catalog.cc:263] Verifying existing consensus state
I20260926 09:29:54.434453 14687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e: Bootstrap starting.
I20260926 09:29:54.440371 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 14618
I20260926 09:29:54.440762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34083
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43703
--webserver_port=44303
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:29:54.443388 14687 log.cc:824] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e: Log is configured to *not* fsync() on all Append() calls
I20260926 09:29:54.445396 14687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:29:54.445727 14687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e: Bootstrap complete.
I20260926 09:29:54.448045 14687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ae0427ed804ab3b3305036c4ea228e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43703 } }
I20260926 09:29:54.448269 14687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40ae0427ed804ab3b3305036c4ea228e, State: Initialized, Role: FOLLOWER
I20260926 09:29:54.448437 14687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ae0427ed804ab3b3305036c4ea228e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43703 } }
I20260926 09:29:54.448508 14687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:29:54.448575 14687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:29:54.448621 14687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 1 FOLLOWER]: Advancing to term 2
I20260926 09:29:54.449717 14687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ae0427ed804ab3b3305036c4ea228e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43703 } }
I20260926 09:29:54.449810 14687 leader_election.cc:302] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40ae0427ed804ab3b3305036c4ea228e; no voters:
I20260926 09:29:54.450038 14687 leader_election.cc:288] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 09:29:54.450073 14691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 2 FOLLOWER]: Leader election won for term 2
I20260926 09:29:54.450182 14691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 2 LEADER]: Becoming Leader. State: Replica: 40ae0427ed804ab3b3305036c4ea228e, State: Running, Role: LEADER
I20260926 09:29:54.450321 14687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:29:54.450342 14691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ae0427ed804ab3b3305036c4ea228e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43703 } }
I20260926 09:29:54.450632 14691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40ae0427ed804ab3b3305036c4ea228e. Latest consensus state: current_term: 2 leader_uuid: "40ae0427ed804ab3b3305036c4ea228e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ae0427ed804ab3b3305036c4ea228e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43703 } } }
I20260926 09:29:54.450639 14692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "40ae0427ed804ab3b3305036c4ea228e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ae0427ed804ab3b3305036c4ea228e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43703 } } }
I20260926 09:29:54.450814 14691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [sys.catalog]: This master's current role is: LEADER
I20260926 09:29:54.450824 14692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [sys.catalog]: This master's current role is: LEADER
I20260926 09:29:54.451074 14700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:29:54.452010 14700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:29:54.452189 14700 catalog_manager.cc:1259] Loaded cluster ID: 4d2a152c303a4b6b976d22dabaffdda6
I20260926 09:29:54.452435 14700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:29:54.452677 14700 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 09:29:54.455472 14700 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 09:29:54.457489 14700 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:29:54.457724 14700 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e: Loaded TSK: 0
I20260926 09:29:54.457962 14700 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 09:29:54.548653 14689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:29:54.548897 14689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:29:54.548935 14689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:29:54.550228 14689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:29:54.550289 14689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:29:54.550334 14689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:29:54.552101 14689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44303
--tserver_master_addrs=127.0.98.62:43703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:29:54.552363 14689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:29:54.552600 14689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:29:54.557730 14717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:54.557791 14714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:54.557720 14715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:54.558125 14689 server_base.cc:1034] running on GCE node
I20260926 09:29:54.558344 14689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:29:54.558620 14689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:29:54.559810 14689 hybrid_clock.cc:648] HybridClock initialized: now 1790414994559779 us; error 46 us; skew 500 ppm
Sep 26 09:29:54 dist-test-slave-dfzm krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:54.562986 14689 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:29:54.563553 14689 webserver.cc:460] Webserver started at http://127.0.98.1:44303/ using document root <none> and password file <none>
I20260926 09:29:54.563829 14689 fs_manager.cc:359] Metadata directory not provided
I20260926 09:29:54.563926 14689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:29:54.565763 14689 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:29:54.566540 14724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:54.566730 14689 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:29:54.566841 14689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "25ee59e37ca2405aa4cc45582ed7ff30"
format_stamp: "Formatted at 2026-09-26 09:29:53 on dist-test-slave-dfzm"
server_key: "d2c0529cde0e230b2c2af1a712a72e07"
server_key_iv: "75998b18b744fb52e44f524e8445d697"
server_key_version: "encryptionkey@0"
I20260926 09:29:54.567196 14689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:29:54.579843 14689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:29:54.582209 14689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:29:54.582384 14689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:29:54.582625 14689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:29:54.583082 14731 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 09:29:54.585038 14689 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 09:29:54.585103 14689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 09:29:54.585166 14689 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 09:29:54.585887 14689 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 09:29:54.585943 14689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260926 09:29:54.586010 14731 tablet_bootstrap.cc:492] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30: Bootstrap starting.
I20260926 09:29:54.593289 14689 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34083
I20260926 09:29:54.593369 14838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34083 every 8 connection(s)
I20260926 09:29:54.593814 14689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:29:54.596489 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 14689
I20260926 09:29:54.596797 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34545
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43703
--webserver_port=39879
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:29:54.596954 14731 log.cc:824] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30: Log is configured to *not* fsync() on all Append() calls
I20260926 09:29:54.600366 14731 tablet_bootstrap.cc:492] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:29:54.600685 14731 tablet_bootstrap.cc:492] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30: Bootstrap complete.
I20260926 09:29:54.601948 14731 ts_tablet_manager.cc:1397] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.015s
I20260926 09:29:54.603375 14731 raft_consensus.cc:348] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } } peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } }
I20260926 09:29:54.603820 14731 raft_consensus.cc:729] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ee59e37ca2405aa4cc45582ed7ff30, State: Initialized, Role: FOLLOWER
I20260926 09:29:54.604030 14731 consensus_queue.cc:260] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } } peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } }
I20260926 09:29:54.604378 14731 ts_tablet_manager.cc:1428] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
Sep 26 09:29:54 dist-test-slave-dfzm krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:29:54.611184 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.594640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.1:47889)
0926 09:29:54.597376 (+ 2736us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:54.597426 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:54.599184 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:54.599664 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:54.599667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:54.599975 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:29:54.600173 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:54.600177 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.601449 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.601450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:54.602851 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:54.602854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.603907 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.603908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:54.604056 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.604716 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:29:54.604727 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:29:54.606628 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:29:54.609145 (+ 2517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:54.609152 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:54.609154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:54.609464 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:54.609813 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:54.609814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:54.609814 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:54.609963 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:54.610505 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:29:54.610512 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:29:54.610669 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 09:29:54.610784 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 09:29:54.610957 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2692,"thread_start_us":95,"threads_started":1}
I20260926 09:29:54.611280 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.594719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47889 (local address 127.0.98.62:43703)
0926 09:29:54.598414 (+ 3695us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:54.598425 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:54.598477 (+ 52us) server_negotiation.cc:407] Connection header received
0926 09:29:54.599450 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:54.599455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:54.599499 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:54.599641 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:54.600291 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.601339 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.602956 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.603685 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.603802 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.606766 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:54.606771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:54.606771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:54.606786 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:54.609040 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:54.609573 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:54.609577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:54.609578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:54.609733 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:54.610051 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:54.610053 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:54.610053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:54.610418 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:54.610518 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:54.610891 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 09:29:54.611053 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3636,"thread_start_us":72,"threads_started":1}
I20260926 09:29:54.612504 14839 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43703
I20260926 09:29:54.612607 14839 heartbeater.cc:464] Registering TS with master...
I20260926 09:29:54.612800 14839 heartbeater.cc:511] Master 127.0.98.62:43703 requested a full tablet report, sending...
I20260926 09:29:54.613559 14651 ts_manager.cc:194] Registered new tserver with Master: 25ee59e37ca2405aa4cc45582ed7ff30 (127.0.98.1:34083)
I20260926 09:29:54.614907 14651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47889
I20260926 09:29:54.616029 14839 heartbeater.cc:503] Master 127.0.98.62:43703 was elected leader, sending a full tablet report...
I20260926 09:29:54.621976 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.614264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34083 (local address 127.0.0.1:55568)
0926 09:29:54.617724 (+ 3460us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:54.617765 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:54.617830 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:54.618151 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:54.618155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:54.618157 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:29:54.618222 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:54.618224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.619474 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.619475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:54.620702 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:54.620705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.621660 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.621661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:54.621762 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.621777 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 09:29:54.621851 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 09:29:54.621898 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3419,"thread_start_us":121,"threads_started":1}
I20260926 09:29:54.622001 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.614541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55568 (local address 127.0.98.1:34083)
0926 09:29:54.617961 (+ 3420us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:54.617965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:54.617974 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:29:54.618007 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:54.618009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:54.618025 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:54.618121 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:29:54.618306 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.619393 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.620807 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.621531 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.621627 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.621683 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:54.621877 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 09:29:54.621930 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3389,"thread_start_us":69,"threads_started":1}
I20260926 09:29:54.623248 14773 tablet_service.cc:1514] Processing DeleteTablet for tablet f68984d78df942eebe911fc5d613d83c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:29:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55568
I20260926 09:29:54.623901 14848 tablet_replica.cc:330] stopping tablet replica
I20260926 09:29:54.624012 14848 raft_consensus.cc:2229] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:29:54.624084 14848 raft_consensus.cc:2258] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:29:54.624367 14848 ts_tablet_manager.cc:1905] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:29:54.626479 14848 ts_tablet_manager.cc:1918] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:29:54.626543 14848 log.cc:1196] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/wal/wals/f68984d78df942eebe911fc5d613d83c
I20260926 09:29:54.626766 14848 ts_tablet_manager.cc:1939] T f68984d78df942eebe911fc5d613d83c P 25ee59e37ca2405aa4cc45582ed7ff30: Deleting consensus metadata
I20260926 09:29:54.627266 14635 catalog_manager.cc:4841] TS 25ee59e37ca2405aa4cc45582ed7ff30 (127.0.98.1:34083): tablet f68984d78df942eebe911fc5d613d83c (table test-table [id=182d175752f94c21aa22240de455401c]) successfully deleted
W20260926 09:29:54.689167 14844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:29:54.689414 14844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:29:54.689456 14844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:29:54.690703 14844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:29:54.690778 14844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:29:54.690824 14844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:29:54.692325 14844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39879
--tserver_master_addrs=127.0.98.62:43703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:29:54.692600 14844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:29:54.692853 14844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:29:54.697875 14854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:54.697899 14853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:54.697955 14844 server_base.cc:1034] running on GCE node
W20260926 09:29:54.697918 14856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:54.698308 14844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:29:54.698565 14844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:29:54.699751 14844 hybrid_clock.cc:648] HybridClock initialized: now 1790414994699739 us; error 30 us; skew 500 ppm
Sep 26 09:29:54 dist-test-slave-dfzm krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:54.702180 14844 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:29:54.702653 14844 webserver.cc:460] Webserver started at http://127.0.98.2:39879/ using document root <none> and password file <none>
I20260926 09:29:54.702881 14844 fs_manager.cc:359] Metadata directory not provided
I20260926 09:29:54.702962 14844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:29:54.704294 14844 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:29:54.704850 14863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:54.705017 14844 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:54.705118 14844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "64f187f1445a4645a8392ab36ed70c00"
format_stamp: "Formatted at 2026-09-26 09:29:53 on dist-test-slave-dfzm"
server_key: "36cbdeed0b5d2ac497e411407b6c2a14"
server_key_iv: "4e4af5dd6f93dab72bcefd5052492bca"
server_key_version: "encryptionkey@0"
I20260926 09:29:54.705469 14844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:29:54.720162 14844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:29:54.722074 14844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:29:54.722235 14844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:29:54.722465 14844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:29:54.722764 14844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:29:54.722819 14844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:54.722869 14844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:29:54.722898 14844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:54.728864 14844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34545
I20260926 09:29:54.728905 14976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34545 every 8 connection(s)
I20260926 09:29:54.729246 14844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:29:54.735010 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 14844
I20260926 09:29:54.735323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41347
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43703
--webserver_port=35389
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:29:54 dist-test-slave-dfzm krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:29:54.739354 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.729992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36009 (local address 127.0.98.62:43703)
0926 09:29:54.730041 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:54.730043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:54.730301 (+ 258us) server_negotiation.cc:407] Connection header received
0926 09:29:54.730761 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:54.730762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:54.730778 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:54.730833 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:54.731753 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.732245 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.733233 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.733424 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.733475 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.736034 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:54.736036 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:54.736036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:54.736044 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:54.737878 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:54.738297 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:54.738298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:54.738299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:54.738365 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:54.738597 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:54.738598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:54.738598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:54.738765 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:54.738836 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:54.739096 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 09:29:54.739205 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:29:54.739440 14979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.730001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.2:36009)
0926 09:29:54.730177 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:54.730213 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:54.730681 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:54.730900 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:54.730904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:54.731480 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:29:54.731676 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:54.731681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.732321 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.732322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:54.733158 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:54.733160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.733494 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.733495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:54.733639 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.734146 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:29:54.734155 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:29:54.735903 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:29:54.737969 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:54.737976 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:54.737978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:54.738217 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:54.738434 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:54.738435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:54.738435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:54.738535 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:54.738830 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:29:54.738834 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:29:54.738961 (+ 127us) client_negotiation.cc:769] Sending connection context
0926 09:29:54.739047 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 09:29:54.739240 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":95,"threads_started":1}
I20260926 09:29:54.740021 14977 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43703
I20260926 09:29:54.740145 14977 heartbeater.cc:464] Registering TS with master...
I20260926 09:29:54.740370 14977 heartbeater.cc:511] Master 127.0.98.62:43703 requested a full tablet report, sending...
I20260926 09:29:54.740818 14651 ts_manager.cc:194] Registered new tserver with Master: 64f187f1445a4645a8392ab36ed70c00 (127.0.98.2:34545)
I20260926 09:29:54.741204 14651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36009
W20260926 09:29:54.818982 14980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:29:54.819185 14980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:29:54.819216 14980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:29:54.820415 14980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:29:54.820459 14980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:29:54.820487 14980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:29:54.821996 14980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35389
--tserver_master_addrs=127.0.98.62:43703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:29:54.822244 14980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:29:54.822479 14980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:29:54.827625 14986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:54.827745 14980 server_base.cc:1034] running on GCE node
W20260926 09:29:54.827647 14988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:54.827625 14985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:54.828122 14980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:29:54.828445 14980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:29:54.829644 14980 hybrid_clock.cc:648] HybridClock initialized: now 1790414994829624 us; error 40 us; skew 500 ppm
Sep 26 09:29:54 dist-test-slave-dfzm krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:54.832235 14980 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:29:54.832703 14980 webserver.cc:460] Webserver started at http://127.0.98.3:35389/ using document root <none> and password file <none>
I20260926 09:29:54.832952 14980 fs_manager.cc:359] Metadata directory not provided
I20260926 09:29:54.833036 14980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:29:54.834475 14980 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:29:54.835130 14995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:54.835327 14980 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:29:54.835453 14980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "4ae513242c614266911e5618e941962f"
format_stamp: "Formatted at 2026-09-26 09:29:54 on dist-test-slave-dfzm"
server_key: "089958be2f78c6b866ff185eb65ce328"
server_key_iv: "3c440b33972e576d7b8626cab3e70954"
server_key_version: "encryptionkey@0"
I20260926 09:29:54.835861 14980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:29:54.872896 14980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:29:54.875164 14980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:29:54.875342 14980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:29:54.875604 14980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:29:54.875922 14980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:29:54.875985 14980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:54.876029 14980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:29:54.876058 14980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:54.882020 14980 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41347
I20260926 09:29:54.882079 15108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41347 every 8 connection(s)
I20260926 09:29:54.882462 14980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:29:54 dist-test-slave-dfzm krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:29:54.891649 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 14980
I20260926 09:29:54.892993 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.883257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47679 (local address 127.0.98.62:43703)
0926 09:29:54.883321 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:54.883324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:54.883617 (+ 293us) server_negotiation.cc:407] Connection header received
0926 09:29:54.884066 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:54.884067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:54.884078 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:54.884137 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:54.885063 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.885584 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.886622 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.886841 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.886919 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.889410 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:54.889412 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:54.889412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:54.889420 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:54.891180 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:54.891655 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:54.891657 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:54.891657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:54.891747 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:54.892157 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:54.892158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:54.892159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:54.892333 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:54.892403 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:54.892770 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 09:29:54.892872 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 09:29:54.893098 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.883259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.3:47679)
0926 09:29:54.883463 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:54.883501 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:54.883979 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:54.884189 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:54.884192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:54.884764 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:29:54.884968 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:54.884972 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.885689 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.885690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:54.886521 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:54.886524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.886937 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.886938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:54.887048 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.887721 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:29:54.887730 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:29:54.889293 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:29:54.891281 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:54.891288 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:54.891290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:54.891553 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:54.891867 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:54.891867 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:54.891868 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:54.892037 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:54.892411 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:29:54.892421 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:29:54.892621 (+ 200us) client_negotiation.cc:769] Sending connection context
0926 09:29:54.892719 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 09:29:54.892875 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":93,"threads_started":1}
I20260926 09:29:54.893697 15109 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43703
I20260926 09:29:54.893831 15109 heartbeater.cc:464] Registering TS with master...
I20260926 09:29:54.894035 15109 heartbeater.cc:511] Master 127.0.98.62:43703 requested a full tablet report, sending...
I20260926 09:29:54.894683 14651 ts_manager.cc:194] Registered new tserver with Master: 4ae513242c614266911e5618e941962f (127.0.98.3:41347)
I20260926 09:29:54.895089 14651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47679
I20260926 09:29:54.898993 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.892128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39348 (local address 127.0.98.62:43703)
0926 09:29:54.892241 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:54.892244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:54.892313 (+ 69us) server_negotiation.cc:407] Connection header received
0926 09:29:54.892408 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:54.892409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:54.892423 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:54.892498 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:54.893192 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.893876 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.894731 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.894977 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.895065 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.896375 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:54.896377 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:54.896377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:54.896384 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:54.897788 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:54.898159 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:54.898161 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:54.898161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:54.898223 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:54.898436 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:54.898438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:54.898438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:54.898564 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:54.898762 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:54.898799 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 09:29:54.898879 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":57,"threads_started":1}
I20260926 09:29:54.899854 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:29:54.903007 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.900239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39362 (local address 127.0.98.62:43703)
0926 09:29:54.900307 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:54.900309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:54.900473 (+ 164us) server_negotiation.cc:407] Connection header received
0926 09:29:54.900506 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:54.900507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:54.900538 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:54.900590 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:29:54.900782 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.901246 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.902214 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.902410 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.902481 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.902653 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:29:54.902752 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:29:54.902814 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:54.902899 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 09:29:54.902926 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 09:29:54.903327 14651 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:29:54.904947 14651 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:29:54.907335 14773 tablet_service.cc:1467] Processing CreateTablet for tablet 9534ed533c1d4bf8bd9690978921b0e0 (DEFAULT_TABLE table=test-table [id=756594d549904bf99aec903fc5011d71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:29:54.907591 14773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9534ed533c1d4bf8bd9690978921b0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:29:54.909294 14731 tablet_bootstrap.cc:492] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30: Bootstrap starting.
I20260926 09:29:54.909713 14731 tablet_bootstrap.cc:654] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30: Neither blocks nor log segments found. Creating new log.
I20260926 09:29:54.910251 14731 tablet_bootstrap.cc:492] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30: No bootstrap required, opened a new log
I20260926 09:29:54.910305 14731 ts_tablet_manager.cc:1397] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 09:29:54.910424 14731 raft_consensus.cc:348] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } } peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } }
I20260926 09:29:54.910490 14731 raft_consensus.cc:374] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:29:54.910528 14731 raft_consensus.cc:729] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ee59e37ca2405aa4cc45582ed7ff30, State: Initialized, Role: FOLLOWER
I20260926 09:29:54.910566 14731 consensus_queue.cc:260] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } } peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } }
I20260926 09:29:54.910661 14731 ts_tablet_manager.cc:1428] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:54.911706 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.907073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41347 (local address 127.0.0.1:60466)
0926 09:29:54.907109 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:54.907120 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:54.907153 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:54.907652 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:54.907653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:54.907654 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:29:54.907700 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:54.907701 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.908884 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.908885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:54.909996 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:54.909998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.911469 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.911471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:54.911529 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.911532 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:29:54.911592 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 09:29:54.911636 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260926 09:29:54.911723 15116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.907253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60466 (local address 127.0.98.3:41347)
0926 09:29:54.907455 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:54.907460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:54.907471 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:29:54.907514 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:54.907517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:54.907535 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:54.907652 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:29:54.907763 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.908786 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.910083 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.911361 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.911468 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.911532 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:54.911607 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 09:29:54.911645 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":77,"threads_started":1}
I20260926 09:29:54.911979 15115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.907166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34545 (local address 127.0.0.1:38632)
0926 09:29:54.907409 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:54.907426 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:54.907484 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:54.907835 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:54.907836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:54.907837 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:29:54.907868 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:54.907869 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.909286 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.909287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:54.910582 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:54.910585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.911703 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.911704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:54.911775 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.911779 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:29:54.911851 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 09:29:54.911897 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":122,"threads_started":1}
I20260926 09:29:54.911991 15117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:54.907465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38632 (local address 127.0.98.2:34545)
0926 09:29:54.907651 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:54.907657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:54.907668 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:29:54.907713 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:54.907716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:54.907732 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:54.907840 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:29:54.907927 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.909110 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.910700 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:54.911598 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:54.911699 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:54.911759 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:54.911866 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 09:29:54.911906 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
I20260926 09:29:54.912761 14911 tablet_service.cc:1467] Processing CreateTablet for tablet 9534ed533c1d4bf8bd9690978921b0e0 (DEFAULT_TABLE table=test-table [id=756594d549904bf99aec903fc5011d71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:29:54.912787 15043 tablet_service.cc:1467] Processing CreateTablet for tablet 9534ed533c1d4bf8bd9690978921b0e0 (DEFAULT_TABLE table=test-table [id=756594d549904bf99aec903fc5011d71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:29:54.913097 14911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9534ed533c1d4bf8bd9690978921b0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:29:54.913112 15043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9534ed533c1d4bf8bd9690978921b0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:29:54.916041 15119 tablet_bootstrap.cc:492] T 9534ed533c1d4bf8bd9690978921b0e0 P 64f187f1445a4645a8392ab36ed70c00: Bootstrap starting.
I20260926 09:29:54.916148 15120 tablet_bootstrap.cc:492] T 9534ed533c1d4bf8bd9690978921b0e0 P 4ae513242c614266911e5618e941962f: Bootstrap starting.
I20260926 09:29:54.916684 15119 tablet_bootstrap.cc:654] T 9534ed533c1d4bf8bd9690978921b0e0 P 64f187f1445a4645a8392ab36ed70c00: Neither blocks nor log segments found. Creating new log.
I20260926 09:29:54.916797 15120 tablet_bootstrap.cc:654] T 9534ed533c1d4bf8bd9690978921b0e0 P 4ae513242c614266911e5618e941962f: Neither blocks nor log segments found. Creating new log.
I20260926 09:29:54.917209 15119 log.cc:824] T 9534ed533c1d4bf8bd9690978921b0e0 P 64f187f1445a4645a8392ab36ed70c00: Log is configured to *not* fsync() on all Append() calls
I20260926 09:29:54.917362 15120 log.cc:824] T 9534ed533c1d4bf8bd9690978921b0e0 P 4ae513242c614266911e5618e941962f: Log is configured to *not* fsync() on all Append() calls
I20260926 09:29:54.918063 15119 tablet_bootstrap.cc:492] T 9534ed533c1d4bf8bd9690978921b0e0 P 64f187f1445a4645a8392ab36ed70c00: No bootstrap required, opened a new log
I20260926 09:29:54.918087 15120 tablet_bootstrap.cc:492] T 9534ed533c1d4bf8bd9690978921b0e0 P 4ae513242c614266911e5618e941962f: No bootstrap required, opened a new log
I20260926 09:29:54.918155 15119 ts_tablet_manager.cc:1397] T 9534ed533c1d4bf8bd9690978921b0e0 P 64f187f1445a4645a8392ab36ed70c00: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:29:54.918144 15120 ts_tablet_manager.cc:1397] T 9534ed533c1d4bf8bd9690978921b0e0 P 4ae513242c614266911e5618e941962f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 09:29:54.919621 15120 raft_consensus.cc:348] T 9534ed533c1d4bf8bd9690978921b0e0 P 4ae513242c614266911e5618e941962f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } } peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } }
I20260926 09:29:54.919763 15120 raft_consensus.cc:374] T 9534ed533c1d4bf8bd9690978921b0e0 P 4ae513242c614266911e5618e941962f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:29:54.919785 15120 raft_consensus.cc:729] T 9534ed533c1d4bf8bd9690978921b0e0 P 4ae513242c614266911e5618e941962f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae513242c614266911e5618e941962f, State: Initialized, Role: FOLLOWER
I20260926 09:29:54.919945 15120 consensus_queue.cc:260] T 9534ed533c1d4bf8bd9690978921b0e0 P 4ae513242c614266911e5618e941962f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } } peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } }
I20260926 09:29:54.919925 15119 raft_consensus.cc:348] T 9534ed533c1d4bf8bd9690978921b0e0 P 64f187f1445a4645a8392ab36ed70c00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } } peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } }
I20260926 09:29:54.920147 15119 raft_consensus.cc:374] T 9534ed533c1d4bf8bd9690978921b0e0 P 64f187f1445a4645a8392ab36ed70c00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:29:54.920210 15119 raft_consensus.cc:729] T 9534ed533c1d4bf8bd9690978921b0e0 P 64f187f1445a4645a8392ab36ed70c00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f187f1445a4645a8392ab36ed70c00, State: Initialized, Role: FOLLOWER
I20260926 09:29:54.920240 15120 ts_tablet_manager.cc:1428] T 9534ed533c1d4bf8bd9690978921b0e0 P 4ae513242c614266911e5618e941962f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 09:29:54.920248 15109 heartbeater.cc:503] Master 127.0.98.62:43703 was elected leader, sending a full tablet report...
I20260926 09:29:54.920339 15119 consensus_queue.cc:260] T 9534ed533c1d4bf8bd9690978921b0e0 P 64f187f1445a4645a8392ab36ed70c00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } } peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } }
I20260926 09:29:54.920624 15119 ts_tablet_manager.cc:1428] T 9534ed533c1d4bf8bd9690978921b0e0 P 64f187f1445a4645a8392ab36ed70c00: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 09:29:54.920748 14977 heartbeater.cc:503] Master 127.0.98.62:43703 was elected leader, sending a full tablet report...
W20260926 09:29:54.979821 14978 tablet.cc:2369] T 9534ed533c1d4bf8bd9690978921b0e0 P 64f187f1445a4645a8392ab36ed70c00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:29:55.031211 14845 raft_consensus.cc:482] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:29:55.031363 14845 raft_consensus.cc:504] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } } peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } }
I20260926 09:29:55.031757 14845 leader_election.cc:288] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ae513242c614266911e5618e941962f (127.0.98.3:41347), 64f187f1445a4645a8392ab36ed70c00 (127.0.98.2:34545)
I20260926 09:29:55.034955 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:55.032088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41347 (local address 127.0.98.1:35503)
0926 09:29:55.032160 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:55.032175 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:55.032249 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:55.032441 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:55.032442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:55.032444 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:29:55.032496 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:55.032498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:55.033222 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:55.033223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:55.034090 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:55.034093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:55.034655 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:55.034656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:55.034731 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:55.034735 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:29:55.034811 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 09:29:55.034858 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 09:29:55.034940 15116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:55.032165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35503 (local address 127.0.98.3:41347)
0926 09:29:55.032242 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:55.032245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:55.032258 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:29:55.032313 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:55.032314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:55.032327 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:55.032383 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:29:55.032572 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:55.033135 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:55.034184 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:55.034567 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:55.034656 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:55.034667 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:55.034830 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 09:29:55.034871 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 09:29:55.035318 15117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:55.032392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51791 (local address 127.0.98.2:34545)
0926 09:29:55.032467 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:55.032469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:55.032481 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:29:55.032550 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:55.032551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:55.032564 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:55.032639 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:29:55.033083 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:55.033538 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:55.034558 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:55.034924 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:55.035011 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:55.035032 (+ 21us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:55.035199 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 09:29:55.035247 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:29:55.035441 15063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9534ed533c1d4bf8bd9690978921b0e0" candidate_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ae513242c614266911e5618e941962f" is_pre_election: true
I20260926 09:29:55.035568 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:55.032223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34545 (local address 127.0.98.1:51791)
0926 09:29:55.032435 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:55.032444 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:55.032494 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:55.032656 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:55.032657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:55.032658 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:29:55.032997 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:55.032999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:55.033698 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:55.033699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:55.034448 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:55.034451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:55.035038 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:55.035039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:55.035111 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:55.035115 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:29:55.035428 (+ 313us) client_negotiation.cc:240] Negotiation successful
0926 09:29:55.035477 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":157,"threads_started":1}
I20260926 09:29:55.035651 15063 raft_consensus.cc:2454] T 9534ed533c1d4bf8bd9690978921b0e0 P 4ae513242c614266911e5618e941962f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25ee59e37ca2405aa4cc45582ed7ff30 in term 0.
I20260926 09:29:55.035893 14727 leader_election.cc:302] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25ee59e37ca2405aa4cc45582ed7ff30, 4ae513242c614266911e5618e941962f; no voters:
I20260926 09:29:55.035877 14931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9534ed533c1d4bf8bd9690978921b0e0" candidate_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f187f1445a4645a8392ab36ed70c00" is_pre_election: true
I20260926 09:29:55.036036 14931 raft_consensus.cc:2454] T 9534ed533c1d4bf8bd9690978921b0e0 P 64f187f1445a4645a8392ab36ed70c00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25ee59e37ca2405aa4cc45582ed7ff30 in term 0.
I20260926 09:29:55.036047 14845 raft_consensus.cc:2781] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:29:55.036154 14845 raft_consensus.cc:482] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:29:55.036187 14845 raft_consensus.cc:3037] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:29:55.036917 14845 raft_consensus.cc:504] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } } peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } }
I20260926 09:29:55.037093 14845 leader_election.cc:288] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [CANDIDATE]: Term 1 election: Requested vote from peers 4ae513242c614266911e5618e941962f (127.0.98.3:41347), 64f187f1445a4645a8392ab36ed70c00 (127.0.98.2:34545)
I20260926 09:29:55.037284 15063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9534ed533c1d4bf8bd9690978921b0e0" candidate_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ae513242c614266911e5618e941962f"
I20260926 09:29:55.037333 14931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9534ed533c1d4bf8bd9690978921b0e0" candidate_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f187f1445a4645a8392ab36ed70c00"
I20260926 09:29:55.037366 15063 raft_consensus.cc:3037] T 9534ed533c1d4bf8bd9690978921b0e0 P 4ae513242c614266911e5618e941962f [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:29:55.037448 14931 raft_consensus.cc:3037] T 9534ed533c1d4bf8bd9690978921b0e0 P 64f187f1445a4645a8392ab36ed70c00 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:29:55.038146 15063 raft_consensus.cc:2454] T 9534ed533c1d4bf8bd9690978921b0e0 P 4ae513242c614266911e5618e941962f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25ee59e37ca2405aa4cc45582ed7ff30 in term 1.
I20260926 09:29:55.038223 14931 raft_consensus.cc:2454] T 9534ed533c1d4bf8bd9690978921b0e0 P 64f187f1445a4645a8392ab36ed70c00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25ee59e37ca2405aa4cc45582ed7ff30 in term 1.
I20260926 09:29:55.038379 14727 leader_election.cc:302] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25ee59e37ca2405aa4cc45582ed7ff30, 64f187f1445a4645a8392ab36ed70c00; no voters:
I20260926 09:29:55.038511 14845 raft_consensus.cc:2781] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:29:55.038682 14845 raft_consensus.cc:686] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 LEADER]: Becoming Leader. State: Replica: 25ee59e37ca2405aa4cc45582ed7ff30, State: Running, Role: LEADER
I20260926 09:29:55.038766 14845 consensus_queue.cc:237] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } } peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } }
I20260926 09:29:55.039677 14650 catalog_manager.cc:5495] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25ee59e37ca2405aa4cc45582ed7ff30 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } health_report { overall_health: HEALTHY } } }
I20260926 09:29:55.056718 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:55.054357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55584 (local address 127.0.98.1:34083)
0926 09:29:55.054425 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:55.054427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:55.054435 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:29:55.054492 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:55.054493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:55.054512 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:55.054571 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:29:55.054717 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:55.055193 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:55.055945 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:55.056125 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:55.056175 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:55.056349 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:29:55.056446 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:29:55.056508 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:55.056550 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 09:29:55.056581 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 09:29:55.058400 15063 raft_consensus.cc:1261] T 9534ed533c1d4bf8bd9690978921b0e0 P 4ae513242c614266911e5618e941962f [term 1 FOLLOWER]: Refusing update from remote peer 25ee59e37ca2405aa4cc45582ed7ff30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:29:55.058442 14931 raft_consensus.cc:1261] T 9534ed533c1d4bf8bd9690978921b0e0 P 64f187f1445a4645a8392ab36ed70c00 [term 1 FOLLOWER]: Refusing update from remote peer 25ee59e37ca2405aa4cc45582ed7ff30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:29:55.058753 15126 consensus_queue.cc:1035] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:29:55.058878 14845 consensus_queue.cc:1035] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:29:55.061694 15131 mvcc.cc:205] Tried to move back new op lower bound from 7333539819756240896 to 7333539819679186944. Current Snapshot: MvccSnapshot[applied={T|T < 7333539819756240896}]
I20260926 09:29:55.062384 15130 mvcc.cc:205] Tried to move back new op lower bound from 7333539819756240896 to 7333539819679186944. Current Snapshot: MvccSnapshot[applied={T|T < 7333539819756240896}]
I20260926 09:29:55.063746 14650 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:29:55.063853 14650 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:29:55.064627 14650 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e: Sending DeleteTablet for 3 replicas of tablet 9534ed533c1d4bf8bd9690978921b0e0
I20260926 09:29:55.064919 14773 tablet_service.cc:1514] Processing DeleteTablet for tablet 9534ed533c1d4bf8bd9690978921b0e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:29:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55568
I20260926 09:29:55.065021 15043 tablet_service.cc:1514] Processing DeleteTablet for tablet 9534ed533c1d4bf8bd9690978921b0e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:29:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60466
I20260926 09:29:55.065093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 14689
I20260926 09:29:55.065074 14911 tablet_service.cc:1514] Processing DeleteTablet for tablet 9534ed533c1d4bf8bd9690978921b0e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:29:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38632
I20260926 09:29:55.065245 14848 tablet_replica.cc:330] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30: stopping tablet replica
I20260926 09:29:55.065295 14848 raft_consensus.cc:2229] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:29:55.065362 14848 raft_consensus.cc:2258] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:29:55.065940 15140 tablet_replica.cc:330] T 9534ed533c1d4bf8bd9690978921b0e0 P 64f187f1445a4645a8392ab36ed70c00: stopping tablet replica
I20260926 09:29:55.066071 15140 raft_consensus.cc:2229] T 9534ed533c1d4bf8bd9690978921b0e0 P 64f187f1445a4645a8392ab36ed70c00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:29:55.066201 15140 raft_consensus.cc:2258] T 9534ed533c1d4bf8bd9690978921b0e0 P 64f187f1445a4645a8392ab36ed70c00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:29:55.066627 15140 ts_tablet_manager.cc:1905] T 9534ed533c1d4bf8bd9690978921b0e0 P 64f187f1445a4645a8392ab36ed70c00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:29:55.068146 15140 ts_tablet_manager.cc:1918] T 9534ed533c1d4bf8bd9690978921b0e0 P 64f187f1445a4645a8392ab36ed70c00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:29:55.068239 15140 log.cc:1196] T 9534ed533c1d4bf8bd9690978921b0e0 P 64f187f1445a4645a8392ab36ed70c00: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/wal/wals/9534ed533c1d4bf8bd9690978921b0e0
I20260926 09:29:55.068557 15140 ts_tablet_manager.cc:1939] T 9534ed533c1d4bf8bd9690978921b0e0 P 64f187f1445a4645a8392ab36ed70c00: Deleting consensus metadata
I20260926 09:29:55.068979 14637 catalog_manager.cc:4841] TS 64f187f1445a4645a8392ab36ed70c00 (127.0.98.2:34545): tablet 9534ed533c1d4bf8bd9690978921b0e0 (table test-table [id=756594d549904bf99aec903fc5011d71]) successfully deleted
I20260926 09:29:55.069728 15139 tablet_replica.cc:330] T 9534ed533c1d4bf8bd9690978921b0e0 P 4ae513242c614266911e5618e941962f: stopping tablet replica
I20260926 09:29:55.069857 15139 raft_consensus.cc:2229] T 9534ed533c1d4bf8bd9690978921b0e0 P 4ae513242c614266911e5618e941962f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:29:55.070020 15139 raft_consensus.cc:2258] T 9534ed533c1d4bf8bd9690978921b0e0 P 4ae513242c614266911e5618e941962f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:29:55.070447 15139 ts_tablet_manager.cc:1905] T 9534ed533c1d4bf8bd9690978921b0e0 P 4ae513242c614266911e5618e941962f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 09:29:55.071347 14635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34083 (error 108)
I20260926 09:29:55.071632 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 14844
I20260926 09:29:55.072043 15139 ts_tablet_manager.cc:1918] T 9534ed533c1d4bf8bd9690978921b0e0 P 4ae513242c614266911e5618e941962f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:29:55.072113 15139 log.cc:1196] T 9534ed533c1d4bf8bd9690978921b0e0 P 4ae513242c614266911e5618e941962f: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/9534ed533c1d4bf8bd9690978921b0e0
I20260926 09:29:55.072315 15115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:55.071796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34083 (local address 127.0.0.1:55596)
0926 09:29:55.072241 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:55.072281 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423}
I20260926 09:29:55.072439 15139 ts_tablet_manager.cc:1939] T 9534ed533c1d4bf8bd9690978921b0e0 P 4ae513242c614266911e5618e941962f: Deleting consensus metadata
W20260926 09:29:55.072429 14635 catalog_manager.cc:4584] TS 25ee59e37ca2405aa4cc45582ed7ff30 (127.0.98.1:34083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9534ed533c1d4bf8bd9690978921b0e0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34083: connect: Connection refused (error 111)
I20260926 09:29:55.072881 14637 catalog_manager.cc:4841] TS 4ae513242c614266911e5618e941962f (127.0.98.3:41347): tablet 9534ed533c1d4bf8bd9690978921b0e0 (table test-table [id=756594d549904bf99aec903fc5011d71]) successfully deleted
I20260926 09:29:55.077682 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 14980
I20260926 09:29:55.082247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 14618
I20260926 09:29:55.087795 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43703
--webserver_interface=127.0.98.62
--webserver_port=40341
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:29:55.168903 15142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:29:55.169127 15142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:29:55.169152 15142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:29:55.170396 15142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:29:55.170454 15142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:29:55.170468 15142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:29:55.170481 15142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:29:55.170493 15142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:29:55.171993 15142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:29:55.172248 15142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:29:55.172516 15142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:29:55.177390 15150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:55.177448 15147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:55.177402 15148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:55.177654 15142 server_base.cc:1034] running on GCE node
I20260926 09:29:55.177878 15142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:29:55.178126 15142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:29:55.179266 15142 hybrid_clock.cc:648] HybridClock initialized: now 1790414995179255 us; error 29 us; skew 500 ppm
Sep 26 09:29:55 dist-test-slave-dfzm krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:55.186743 15142 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:29:55.187224 15142 webserver.cc:460] Webserver started at http://127.0.98.62:40341/ using document root <none> and password file <none>
I20260926 09:29:55.187472 15142 fs_manager.cc:359] Metadata directory not provided
I20260926 09:29:55.187526 15142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:29:55.188967 15142 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:29:55.189687 15157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:55.189842 15142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:55.189940 15142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "40ae0427ed804ab3b3305036c4ea228e"
format_stamp: "Formatted at 2026-09-26 09:29:53 on dist-test-slave-dfzm"
server_key: "12a495eabcfd8026e46757e694d525cd"
server_key_iv: "ce9a32e3d5f1f745b801bb13284c2a51"
server_key_version: "encryptionkey@0"
I20260926 09:29:55.190300 15142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:29:55.221908 15142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:29:55.223899 15142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:29:55.224020 15142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:29:55.228022 15142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43703
I20260926 09:29:55.228078 15209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43703 every 8 connection(s)
I20260926 09:29:55.228468 15142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:29:55.230113 15210 sys_catalog.cc:263] Verifying existing consensus state
I20260926 09:29:55.230779 15210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e: Bootstrap starting.
I20260926 09:29:55.233124 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 15142
I20260926 09:29:55.233467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34083
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43703
--webserver_port=44303
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:29:55.239662 15210 log.cc:824] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e: Log is configured to *not* fsync() on all Append() calls
I20260926 09:29:55.242046 15210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:29:55.242370 15210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e: Bootstrap complete.
I20260926 09:29:55.244632 15210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ae0427ed804ab3b3305036c4ea228e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43703 } }
I20260926 09:29:55.244978 15210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40ae0427ed804ab3b3305036c4ea228e, State: Initialized, Role: FOLLOWER
I20260926 09:29:55.245160 15210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ae0427ed804ab3b3305036c4ea228e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43703 } }
I20260926 09:29:55.245235 15210 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:29:55.245313 15210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:29:55.245383 15210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 2 FOLLOWER]: Advancing to term 3
I20260926 09:29:55.246461 15210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ae0427ed804ab3b3305036c4ea228e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43703 } }
I20260926 09:29:55.246549 15210 leader_election.cc:302] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40ae0427ed804ab3b3305036c4ea228e; no voters:
I20260926 09:29:55.246771 15210 leader_election.cc:288] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 09:29:55.246809 15214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 3 FOLLOWER]: Leader election won for term 3
I20260926 09:29:55.246927 15214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [term 3 LEADER]: Becoming Leader. State: Replica: 40ae0427ed804ab3b3305036c4ea228e, State: Running, Role: LEADER
I20260926 09:29:55.247051 15210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:29:55.247192 15214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ae0427ed804ab3b3305036c4ea228e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43703 } }
I20260926 09:29:55.247494 15215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "40ae0427ed804ab3b3305036c4ea228e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ae0427ed804ab3b3305036c4ea228e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43703 } } }
I20260926 09:29:55.247699 15215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [sys.catalog]: This master's current role is: LEADER
I20260926 09:29:55.247493 15214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40ae0427ed804ab3b3305036c4ea228e. Latest consensus state: current_term: 3 leader_uuid: "40ae0427ed804ab3b3305036c4ea228e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ae0427ed804ab3b3305036c4ea228e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43703 } } }
I20260926 09:29:55.247877 15214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ae0427ed804ab3b3305036c4ea228e [sys.catalog]: This master's current role is: LEADER
I20260926 09:29:55.248000 15223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:29:55.248922 15223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:29:55.249091 15223 catalog_manager.cc:1259] Loaded cluster ID: 4d2a152c303a4b6b976d22dabaffdda6
I20260926 09:29:55.249311 15223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 09:29:55.249531 15223 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff0707309ad google::LogMessage::Fail() at ??:0
@ 0x7ff07073388a google::LogMessage::SendToLog() at ??:0
@ 0x7ff0707304de google::LogMessage::Flush() at ??:0
@ 0x7ff070731a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff0786510f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ff07865a8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff0725fe0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff0725f3954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff0734576db start_thread at ??:0
@ 0x7ff06f22e61f clone at ??:0
W20260926 09:29:55.332525 15212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:29:55.332727 15212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:29:55.332760 15212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:29:55.333995 15212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:29:55.334036 15212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:29:55.334064 15212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:29:55.335527 15212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44303
--tserver_master_addrs=127.0.98.62:43703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:29:55.335757 15212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:29:55.336009 15212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:29:55.341050 15246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:55.341063 15249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:55.341336 15212 server_base.cc:1034] running on GCE node
W20260926 09:29:55.341179 15247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:55.341655 15212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:29:55.341951 15212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:29:55.343125 15212 hybrid_clock.cc:648] HybridClock initialized: now 1790414995343106 us; error 28 us; skew 500 ppm
Sep 26 09:29:55 dist-test-slave-dfzm krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:55.345800 15212 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:29:55.346302 15212 webserver.cc:460] Webserver started at http://127.0.98.1:44303/ using document root <none> and password file <none>
I20260926 09:29:55.346534 15212 fs_manager.cc:359] Metadata directory not provided
I20260926 09:29:55.346577 15212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:29:55.348026 15212 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:29:55.348811 15256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:55.349040 15212 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:29:55.349164 15212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "25ee59e37ca2405aa4cc45582ed7ff30"
format_stamp: "Formatted at 2026-09-26 09:29:53 on dist-test-slave-dfzm"
server_key: "d2c0529cde0e230b2c2af1a712a72e07"
server_key_iv: "75998b18b744fb52e44f524e8445d697"
server_key_version: "encryptionkey@0"
I20260926 09:29:55.349568 15212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:29:55.362800 15212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:29:55.365090 15212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:29:55.365275 15212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:29:55.365526 15212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:29:55.366089 15263 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 09:29:55.367547 15212 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 09:29:55.367604 15212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 09:29:55.367630 15212 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 09:29:55.368317 15212 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 09:29:55.368362 15212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 09:29:55.368407 15263 tablet_bootstrap.cc:492] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30: Bootstrap starting.
I20260926 09:29:55.376078 15212 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34083
I20260926 09:29:55.376273 15370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34083 every 8 connection(s)
I20260926 09:29:55.376627 15212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:29:55.379194 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 15212
I20260926 09:29:55.379492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34545
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43703
--webserver_port=39879
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:29:55.379776 15263 log.cc:824] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30: Log is configured to *not* fsync() on all Append() calls
I20260926 09:29:55.382336 15263 tablet_bootstrap.cc:492] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:29:55.382639 15263 tablet_bootstrap.cc:492] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30: Bootstrap complete.
I20260926 09:29:55.383810 15263 ts_tablet_manager.cc:1397] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
I20260926 09:29:55.385672 15263 raft_consensus.cc:348] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } } peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } }
I20260926 09:29:55.386101 15263 raft_consensus.cc:729] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ee59e37ca2405aa4cc45582ed7ff30, State: Initialized, Role: FOLLOWER
I20260926 09:29:55.386260 15263 consensus_queue.cc:260] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } } peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } }
I20260926 09:29:55.386564 15263 ts_tablet_manager.cc:1428] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
W20260926 09:29:55.448393 15373 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:29:55.377367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.1:52817)
0926 09:29:55.377688 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:55.377733 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:55.378365 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:55.448161 (+ 69796us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":105,"threads_started":1}
W20260926 09:29:55.448709 15371 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43703 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43703: Client connection negotiation failed: client connection to 127.0.98.62:43703: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 09:29:55.449054 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:55.448932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.1:39577)
0926 09:29:55.448986 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:55.449026 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 09:29:55.449434 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:55.449353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.1:49773)
0926 09:29:55.449398 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:55.449421 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 09:29:55.449647 15371 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 09:29:55.466547 15375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:29:55.466746 15375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:29:55.466770 15375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:29:55.467926 15375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:29:55.467969 15375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:29:55.467996 15375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:29:55.469462 15375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39879
--tserver_master_addrs=127.0.98.62:43703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:29:55.469728 15375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:29:55.469969 15375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:29:55.474922 15381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:55.474911 15384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:55.474898 15382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:55.475186 15375 server_base.cc:1034] running on GCE node
I20260926 09:29:55.475430 15375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:29:55.475706 15375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:29:55.476871 15375 hybrid_clock.cc:648] HybridClock initialized: now 1790414995476864 us; error 39 us; skew 500 ppm
Sep 26 09:29:55 dist-test-slave-dfzm krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:55.479385 15375 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:29:55.479822 15375 webserver.cc:460] Webserver started at http://127.0.98.2:39879/ using document root <none> and password file <none>
I20260926 09:29:55.480072 15375 fs_manager.cc:359] Metadata directory not provided
I20260926 09:29:55.480141 15375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:29:55.481431 15375 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:29:55.482064 15391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:55.482220 15375 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:55.482324 15375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "64f187f1445a4645a8392ab36ed70c00"
format_stamp: "Formatted at 2026-09-26 09:29:53 on dist-test-slave-dfzm"
server_key: "36cbdeed0b5d2ac497e411407b6c2a14"
server_key_iv: "4e4af5dd6f93dab72bcefd5052492bca"
server_key_version: "encryptionkey@0"
I20260926 09:29:55.482682 15375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:29:55.495184 15375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:29:55.497651 15375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:29:55.497819 15375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:29:55.498057 15375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:29:55.498359 15375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:29:55.498413 15375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:55.498458 15375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:29:55.498494 15375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:55.504463 15375 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34545
I20260926 09:29:55.504549 15504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34545 every 8 connection(s)
I20260926 09:29:55.504906 15375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:29:55.506031 15507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:55.505572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.2:49669)
0926 09:29:55.505768 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:55.505877 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":76,"threads_started":1}
W20260926 09:29:55.506400 15505 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43703 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43703: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
I20260926 09:29:55.506707 15507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:55.506594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.2:51407)
0926 09:29:55.506648 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:55.506681 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 09:29:55.506744 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 15375
I20260926 09:29:55.507015 15507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:55.506938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.2:43847)
0926 09:29:55.506963 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:55.506988 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
I20260926 09:29:55.507030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41347
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43703
--webserver_port=35389
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:29:55.507118 15505 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 09:29:55.588258 15508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:29:55.588511 15508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:29:55.588553 15508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:29:55.589890 15508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:29:55.589948 15508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:29:55.589998 15508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:29:55.591593 15508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35389
--tserver_master_addrs=127.0.98.62:43703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:29:55.591864 15508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:29:55.592115 15508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:29:55.597301 15514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:29:55.597281 15513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:55.597319 15508 server_base.cc:1034] running on GCE node
W20260926 09:29:55.597273 15516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:29:55.597749 15508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:29:55.598021 15508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:29:55.599203 15508 hybrid_clock.cc:648] HybridClock initialized: now 1790414995599187 us; error 27 us; skew 500 ppm
Sep 26 09:29:55 dist-test-slave-dfzm krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:29:55.601733 15508 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:29:55.602139 15508 webserver.cc:460] Webserver started at http://127.0.98.3:35389/ using document root <none> and password file <none>
I20260926 09:29:55.602332 15508 fs_manager.cc:359] Metadata directory not provided
I20260926 09:29:55.602375 15508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:29:55.603665 15508 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:29:55.604223 15523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:55.604390 15508 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:29:55.604494 15508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "4ae513242c614266911e5618e941962f"
format_stamp: "Formatted at 2026-09-26 09:29:54 on dist-test-slave-dfzm"
server_key: "089958be2f78c6b866ff185eb65ce328"
server_key_iv: "3c440b33972e576d7b8626cab3e70954"
server_key_version: "encryptionkey@0"
I20260926 09:29:55.604830 15508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:29:55.622104 15508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:29:55.624502 15508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:29:55.624670 15508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:29:55.624917 15508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:29:55.625229 15508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:29:55.625283 15508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:55.625325 15508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:29:55.625353 15508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:29:55.632308 15508 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41347
I20260926 09:29:55.632391 15636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41347 every 8 connection(s)
I20260926 09:29:55.632824 15508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:29:55.634181 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:55.633748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.3:52263)
0926 09:29:55.633927 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:55.634014 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":104,"threads_started":1}
W20260926 09:29:55.634634 15637 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43703 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43703: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
I20260926 09:29:55.634990 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:55.634881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.3:53863)
0926 09:29:55.634944 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:55.634973 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 09:29:55.635279 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:55.635224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.3:49663)
0926 09:29:55.635248 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:55.635267 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260926 09:29:55.635412 15637 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 09:29:55.641986 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 15508
I20260926 09:29:56.450246 15641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:56.450015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.1:35651)
0926 09:29:56.450147 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:56.450188 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260926 09:29:56.507675 15642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:56.507458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.2:53739)
0926 09:29:56.507576 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:56.507623 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260926 09:29:56.516294 15643 raft_consensus.cc:482] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:29:56.516463 15643 raft_consensus.cc:504] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } } peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } }
I20260926 09:29:56.516826 15643 leader_election.cc:288] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ae513242c614266911e5618e941962f (127.0.98.3:41347), 64f187f1445a4645a8392ab36ed70c00 (127.0.98.2:34545)
Sep 26 09:29:56 dist-test-slave-dfzm krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 26 09:29:56 dist-test-slave-dfzm krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260926 09:29:56.528038 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:56.517123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41933 (local address 127.0.98.3:41347)
0926 09:29:56.517316 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:56.517328 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:56.517354 (+ 26us) server_negotiation.cc:407] Connection header received
0926 09:29:56.517847 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:56.517849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:56.517916 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:56.518066 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:56.518801 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:56.519678 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:56.520694 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:56.521189 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:56.521314 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:56.524080 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:56.524085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:56.524085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:56.524099 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:56.526227 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:56.526659 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:56.526661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:56.526662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:56.526752 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:56.527020 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:56.527021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:56.527021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:56.527354 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:56.527446 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:56.527799 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 09:29:56.527946 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
I20260926 09:29:56.528049 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:56.517226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34721 (local address 127.0.98.2:34545)
0926 09:29:56.517328 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 09:29:56.517338 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:29:56.517374 (+ 36us) server_negotiation.cc:407] Connection header received
0926 09:29:56.517855 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:29:56.517858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:29:56.517899 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:29:56.518039 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:29:56.518798 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:56.519660 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:56.520646 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:29:56.521190 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:29:56.521305 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:56.524553 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:29:56.524578 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:29:56.524578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:29:56.524592 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:29:56.526491 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:56.526840 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:56.526842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:56.526843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:56.526928 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:29:56.527121 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:29:56.527122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:29:56.527122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:29:56.527368 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:29:56.527484 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 09:29:56.527799 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 09:29:56.527935 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68,"thread_start_us":49,"threads_started":1}
I20260926 09:29:56.528049 15641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:56.516992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41347 (local address 127.0.98.1:41933)
0926 09:29:56.517048 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:56.517060 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:56.517117 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:56.518089 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:56.518093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:56.518493 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:29:56.518674 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:56.518680 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:56.519801 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:56.519802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:56.520581 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:56.520583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:56.521323 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:56.521324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:56.521443 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:56.522230 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:29:56.522234 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:29:56.523971 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:29:56.526313 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:56.526320 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:56.526322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:56.526575 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:56.526825 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:56.526825 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:56.526826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:56.526935 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:56.527435 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:29:56.527444 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:29:56.527607 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 09:29:56.527727 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 09:29:56.527932 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":96}
I20260926 09:29:56.528276 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:56.517138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34545 (local address 127.0.98.1:34721)
0926 09:29:56.517245 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:56.517254 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:29:56.517295 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:29:56.518090 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:29:56.518093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:29:56.518494 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:29:56.518674 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:56.518680 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:56.519769 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:56.519770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:56.520562 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:29:56.520564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:29:56.521389 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:29:56.521389 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:29:56.521443 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:29:56.522196 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:29:56.522207 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:29:56.524451 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:29:56.526578 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:56.526579 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:56.526579 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:56.526758 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:56.526995 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:29:56.526996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:29:56.526996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:29:56.527060 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:29:56.527435 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:29:56.527444 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:29:56.527607 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 09:29:56.527727 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 09:29:56.527985 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":129,"thread_start_us":66,"threads_started":1}
W20260926 09:29:56.528954 15259 leader_election.cc:341] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ae513242c614266911e5618e941962f (127.0.98.3:41347): Not found: Tablet not found: 9534ed533c1d4bf8bd9690978921b0e0
W20260926 09:29:56.529150 15259 leader_election.cc:341] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 64f187f1445a4645a8392ab36ed70c00 (127.0.98.2:34545): Not found: Tablet not found: 9534ed533c1d4bf8bd9690978921b0e0
I20260926 09:29:56.529202 15259 leader_election.cc:302] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25ee59e37ca2405aa4cc45582ed7ff30; no voters: 4ae513242c614266911e5618e941962f, 64f187f1445a4645a8392ab36ed70c00
I20260926 09:29:56.529331 15643 raft_consensus.cc:2726] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:29:56.636014 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:56.635779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.3:53507)
0926 09:29:56.635914 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:56.635971 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":56,"threads_started":1}
I20260926 09:29:57.450959 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:57.450760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.1:53729)
0926 09:29:57.450875 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:57.450912 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260926 09:29:57.508304 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:57.508099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.2:51919)
0926 09:29:57.508213 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:57.508266 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260926 09:29:57.636708 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:57.636509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.3:55247)
0926 09:29:57.636625 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:57.636665 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260926 09:29:58.451735 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:58.451435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.1:40519)
0926 09:29:58.451613 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:58.451664 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":64,"threads_started":1}
I20260926 09:29:58.475457 15652 raft_consensus.cc:482] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:29:58.475548 15652 raft_consensus.cc:504] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } } peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } }
I20260926 09:29:58.475668 15652 leader_election.cc:288] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ae513242c614266911e5618e941962f (127.0.98.3:41347), 64f187f1445a4645a8392ab36ed70c00 (127.0.98.2:34545)
W20260926 09:29:58.476120 15259 leader_election.cc:341] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 64f187f1445a4645a8392ab36ed70c00 (127.0.98.2:34545): Not found: Tablet not found: 9534ed533c1d4bf8bd9690978921b0e0
W20260926 09:29:58.476200 15259 leader_election.cc:341] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ae513242c614266911e5618e941962f (127.0.98.3:41347): Not found: Tablet not found: 9534ed533c1d4bf8bd9690978921b0e0
I20260926 09:29:58.476284 15259 leader_election.cc:302] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25ee59e37ca2405aa4cc45582ed7ff30; no voters: 4ae513242c614266911e5618e941962f, 64f187f1445a4645a8392ab36ed70c00
I20260926 09:29:58.476415 15652 raft_consensus.cc:2726] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:29:58.509033 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:58.508773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.2:47197)
0926 09:29:58.508953 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:58.508990 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":119,"threads_started":1}
I20260926 09:29:58.637316 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:58.637122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.3:33083)
0926 09:29:58.637242 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:58.637275 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260926 09:29:59.452606 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:59.452345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.1:54167)
0926 09:29:59.452518 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:59.452560 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":98,"threads_started":1}
I20260926 09:29:59.509763 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:59.509498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.2:52645)
0926 09:29:59.509677 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:59.509719 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":63,"threads_started":1}
I20260926 09:29:59.637979 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:29:59.637770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.3:44561)
0926 09:29:59.637895 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 09:29:59.637936 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
I20260926 09:30:00.366485 15658 raft_consensus.cc:482] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:30:00.366598 15658 raft_consensus.cc:504] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } } peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } }
I20260926 09:30:00.366793 15658 leader_election.cc:288] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ae513242c614266911e5618e941962f (127.0.98.3:41347), 64f187f1445a4645a8392ab36ed70c00 (127.0.98.2:34545)
W20260926 09:30:00.367170 15259 leader_election.cc:341] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ae513242c614266911e5618e941962f (127.0.98.3:41347): Not found: Tablet not found: 9534ed533c1d4bf8bd9690978921b0e0
W20260926 09:30:00.367344 15259 leader_election.cc:341] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 64f187f1445a4645a8392ab36ed70c00 (127.0.98.2:34545): Not found: Tablet not found: 9534ed533c1d4bf8bd9690978921b0e0
I20260926 09:30:00.367400 15259 leader_election.cc:302] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25ee59e37ca2405aa4cc45582ed7ff30; no voters: 4ae513242c614266911e5618e941962f, 64f187f1445a4645a8392ab36ed70c00
I20260926 09:30:00.367467 15658 raft_consensus.cc:2726] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:30:00.453411 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:00.453145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.1:59007)
0926 09:30:00.453307 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:00.453351 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260926 09:30:00.510470 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:00.510213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.2:57577)
0926 09:30:00.510362 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:00.510416 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260926 09:30:00.638784 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:00.638529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.3:41527)
0926 09:30:00.638687 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:00.638740 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":65,"threads_started":1}
I20260926 09:30:01.454197 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:01.453975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.1:55977)
0926 09:30:01.454102 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:01.454150 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260926 09:30:01.511145 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:01.510936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.2:35233)
0926 09:30:01.511059 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:01.511096 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260926 09:30:01.639559 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:01.639346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.3:43347)
0926 09:30:01.639473 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:01.639512 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":59,"threads_started":1}
I20260926 09:30:02.455065 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:02.454825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.1:33067)
0926 09:30:02.454973 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:02.455018 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
I20260926 09:30:02.512070 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:02.511796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.2:42939)
0926 09:30:02.511924 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:02.511971 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260926 09:30:02.640225 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:02.640007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.3:56295)
0926 09:30:02.640141 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:02.640182 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20260926 09:30:03.455894 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:03.455662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.1:57815)
0926 09:30:03.455808 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:03.455848 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
I20260926 09:30:03.512861 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:03.512594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.2:34277)
0926 09:30:03.512764 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:03.512813 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":109,"threads_started":1}
I20260926 09:30:03.641005 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:03.640769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.3:52265)
0926 09:30:03.640904 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:03.640949 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260926 09:30:04.456646 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:04.456404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.1:47539)
0926 09:30:04.456556 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:04.456599 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":67,"threads_started":1}
I20260926 09:30:04.513571 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:04.513330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.2:56369)
0926 09:30:04.513455 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:04.513504 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260926 09:30:04.641845 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:04.641498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.3:50083)
0926 09:30:04.641764 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:04.641803 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":59,"threads_started":1}
I20260926 09:30:04.676728 15674 raft_consensus.cc:482] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:30:04.676827 15674 raft_consensus.cc:504] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } } peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } }
I20260926 09:30:04.676980 15674 leader_election.cc:288] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ae513242c614266911e5618e941962f (127.0.98.3:41347), 64f187f1445a4645a8392ab36ed70c00 (127.0.98.2:34545)
W20260926 09:30:04.677446 15259 leader_election.cc:341] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ae513242c614266911e5618e941962f (127.0.98.3:41347): Not found: Tablet not found: 9534ed533c1d4bf8bd9690978921b0e0
W20260926 09:30:04.677560 15259 leader_election.cc:341] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 64f187f1445a4645a8392ab36ed70c00 (127.0.98.2:34545): Not found: Tablet not found: 9534ed533c1d4bf8bd9690978921b0e0
I20260926 09:30:04.677623 15259 leader_election.cc:302] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25ee59e37ca2405aa4cc45582ed7ff30; no voters: 4ae513242c614266911e5618e941962f, 64f187f1445a4645a8392ab36ed70c00
I20260926 09:30:04.677697 15674 raft_consensus.cc:2726] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:30:05.457466 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:05.457182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.1:60689)
0926 09:30:05.457352 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:05.457404 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
I20260926 09:30:05.514274 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:05.514076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.2:52139)
0926 09:30:05.514193 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:05.514231 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260926 09:30:05.642565 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:05.642353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.3:59517)
0926 09:30:05.642482 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:05.642519 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260926 09:30:06.458292 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:06.458038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.1:56375)
0926 09:30:06.458198 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:06.458242 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":96,"threads_started":1}
I20260926 09:30:06.514956 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:06.514710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.2:36021)
0926 09:30:06.514865 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:06.514910 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":75,"threads_started":1}
I20260926 09:30:06.643337 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:06.643068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.3:33283)
0926 09:30:06.643245 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:06.643290 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":79,"threads_started":1}
I20260926 09:30:07.325987 15681 raft_consensus.cc:482] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:30:07.326104 15681 raft_consensus.cc:504] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae513242c614266911e5618e941962f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41347 } } peers { permanent_uuid: "64f187f1445a4645a8392ab36ed70c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "25ee59e37ca2405aa4cc45582ed7ff30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34083 } }
I20260926 09:30:07.326299 15681 leader_election.cc:288] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ae513242c614266911e5618e941962f (127.0.98.3:41347), 64f187f1445a4645a8392ab36ed70c00 (127.0.98.2:34545)
W20260926 09:30:07.326694 15259 leader_election.cc:341] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 64f187f1445a4645a8392ab36ed70c00 (127.0.98.2:34545): Not found: Tablet not found: 9534ed533c1d4bf8bd9690978921b0e0
W20260926 09:30:07.326767 15259 leader_election.cc:341] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ae513242c614266911e5618e941962f (127.0.98.3:41347): Not found: Tablet not found: 9534ed533c1d4bf8bd9690978921b0e0
I20260926 09:30:07.326784 15259 leader_election.cc:302] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25ee59e37ca2405aa4cc45582ed7ff30; no voters: 4ae513242c614266911e5618e941962f, 64f187f1445a4645a8392ab36ed70c00
I20260926 09:30:07.326853 15681 raft_consensus.cc:2726] T 9534ed533c1d4bf8bd9690978921b0e0 P 25ee59e37ca2405aa4cc45582ed7ff30 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:30:07.459004 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:07.458791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.1:35229)
0926 09:30:07.458923 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:07.458958 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":57,"threads_started":1}
I20260926 09:30:07.515647 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:07.515428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.2:56075)
0926 09:30:07.515565 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:07.515603 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":59,"threads_started":1}
I20260926 09:30:07.644065 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:07.643809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.3:49681)
0926 09:30:07.643977 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:07.644018 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":76,"threads_started":1}
I20260926 09:30:08.459645 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:08.459436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.1:35689)
0926 09:30:08.459570 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:08.459600 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260926 09:30:08.516294 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:08.516064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.2:49701)
0926 09:30:08.516211 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:08.516249 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260926 09:30:08.644817 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:08.644580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.3:51809)
0926 09:30:08.644726 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:08.644772 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":66,"threads_started":1}
I20260926 09:30:09.460469 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:09.460225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.1:42751)
0926 09:30:09.460375 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:09.460422 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":64,"threads_started":1}
I20260926 09:30:09.517088 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:09.516787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.2:39351)
0926 09:30:09.516959 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:09.517042 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":80,"threads_started":1}
I20260926 09:30:09.645567 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:09.645301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.3:55765)
0926 09:30:09.645480 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:09.645521 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":79,"threads_started":1}
I20260926 09:30:10.461396 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:10.461156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.1:48809)
0926 09:30:10.461306 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:10.461350 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":72,"threads_started":1}
I20260926 09:30:10.517892 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:10.517679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.2:53129)
0926 09:30:10.517806 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:10.517845 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260926 09:30:10.643170 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 15212
I20260926 09:30:10.646463 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:10.646239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.3:55617)
0926 09:30:10.646382 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:10.646419 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260926 09:30:10.647786 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 15375
I20260926 09:30:10.652264 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 15508
2026-09-26T09:30:10Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18396 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:30:10 dist-test-slave-dfzm krb5kdc[15696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:30:10 dist-test-slave-dfzm krb5kdc[15696](info): set up 2 sockets
Sep 26 09:30:10 dist-test-slave-dfzm krb5kdc[15696](info): commencing operation
krb5kdc: starting...
W20260926 09:30:11.685730 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:30:11 dist-test-slave-dfzm krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:30:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:30:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:30:11.816867 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39239
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:30:11.899056 15709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:30:11.899288 15709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:30:11.899333 15709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:30:11.900535 15709 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 09:30:11.900604 15709 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 09:30:11.900627 15709 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 09:30:11.900657 15709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:30:11.900686 15709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:30:11.900715 15709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:30:11.900743 15709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:30:11.900764 15709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:30:11.902374 15709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:30:11.902659 15709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:30:11.902895 15709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:30:11.908164 15717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:30:11.908263 15714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:30:11.908263 15715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:30:11.908243 15709 server_base.cc:1034] running on GCE node
I20260926 09:30:11.908732 15709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:30:11.909027 15709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:30:11.910199 15709 hybrid_clock.cc:648] HybridClock initialized: now 1790415011910193 us; error 38 us; skew 500 ppm
Sep 26 09:30:11 dist-test-slave-dfzm krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:30:11.912678 15709 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:30:11.913148 15709 webserver.cc:460] Webserver started at http://127.0.98.62:42631/ using document root <none> and password file <none>
I20260926 09:30:11.913358 15709 fs_manager.cc:359] Metadata directory not provided
I20260926 09:30:11.913399 15709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:30:11.913488 15709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:30:11.914570 15709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "9a9e0464302f4d018adea0b9aa6ca386"
format_stamp: "Formatted at 2026-09-26 09:30:11 on dist-test-slave-dfzm"
server_key: "00a89eeef900c5a3a023b164d153b862"
server_key_iv: "7e73e20109a3ec5079d2545bc86d59a6"
server_key_version: "encryptionkey@0"
I20260926 09:30:11.914857 15709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "9a9e0464302f4d018adea0b9aa6ca386"
format_stamp: "Formatted at 2026-09-26 09:30:11 on dist-test-slave-dfzm"
server_key: "00a89eeef900c5a3a023b164d153b862"
server_key_iv: "7e73e20109a3ec5079d2545bc86d59a6"
server_key_version: "encryptionkey@0"
I20260926 09:30:11.916162 15709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:30:11.916987 15724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:11.917150 15709 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:30:11.917232 15709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "9a9e0464302f4d018adea0b9aa6ca386"
format_stamp: "Formatted at 2026-09-26 09:30:11 on dist-test-slave-dfzm"
server_key: "00a89eeef900c5a3a023b164d153b862"
server_key_iv: "7e73e20109a3ec5079d2545bc86d59a6"
server_key_version: "encryptionkey@0"
I20260926 09:30:11.917281 15709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:30:11.935316 15709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:30:11.937770 15709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:30:11.937903 15709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:30:11.941730 15709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39239
I20260926 09:30:11.941816 15776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39239 every 8 connection(s)
I20260926 09:30:11.942152 15709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:30:11.943001 15777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:30:11.945331 15777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386: Bootstrap starting.
I20260926 09:30:11.945945 15777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386: Neither blocks nor log segments found. Creating new log.
I20260926 09:30:11.946237 15777 log.cc:824] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386: Log is configured to *not* fsync() on all Append() calls
I20260926 09:30:11.946910 15777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386: No bootstrap required, opened a new log
I20260926 09:30:11.948568 15777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e0464302f4d018adea0b9aa6ca386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39239 } }
I20260926 09:30:11.948741 15777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:30:11.948781 15777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9e0464302f4d018adea0b9aa6ca386, State: Initialized, Role: FOLLOWER
I20260926 09:30:11.948957 15777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e0464302f4d018adea0b9aa6ca386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39239 } }
I20260926 09:30:11.949043 15777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:30:11.949080 15777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:30:11.949132 15777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:30:11.949699 15777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e0464302f4d018adea0b9aa6ca386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39239 } }
I20260926 09:30:11.949819 15777 leader_election.cc:302] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a9e0464302f4d018adea0b9aa6ca386; no voters:
I20260926 09:30:11.950006 15777 leader_election.cc:288] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:30:11.950035 15780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:30:11.950261 15777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:30:11.950260 15780 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 1 LEADER]: Becoming Leader. State: Replica: 9a9e0464302f4d018adea0b9aa6ca386, State: Running, Role: LEADER
I20260926 09:30:11.950394 15780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e0464302f4d018adea0b9aa6ca386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39239 } }
I20260926 09:30:11.950799 15781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a9e0464302f4d018adea0b9aa6ca386" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e0464302f4d018adea0b9aa6ca386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39239 } } }
I20260926 09:30:11.950812 15782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a9e0464302f4d018adea0b9aa6ca386. Latest consensus state: current_term: 1 leader_uuid: "9a9e0464302f4d018adea0b9aa6ca386" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e0464302f4d018adea0b9aa6ca386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39239 } } }
I20260926 09:30:11.950949 15781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [sys.catalog]: This master's current role is: LEADER
I20260926 09:30:11.951009 15782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [sys.catalog]: This master's current role is: LEADER
I20260926 09:30:11.951380 15789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:30:11.951895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 15709
I20260926 09:30:11.951898 15789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:30:11.951999 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:30:11.952235 392 external_mini_cluster.cc:1428] Setting key 2a82b4c4d32aef898a099b4efb799248
I20260926 09:30:11.954180 15789 catalog_manager.cc:1347] Generated new cluster ID: 56188d79318a40be977d4454fc5f2b8b
I20260926 09:30:11.954232 15789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:30:11 dist-test-slave-dfzm krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:30:11.966876 15789 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:30:11.967594 15789 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:30:11.967929 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:11.952808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53150 (local address 127.0.98.62:39239)
0926 09:30:11.953878 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:11.953887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:11.953919 (+ 32us) server_negotiation.cc:407] Connection header received
0926 09:30:11.954043 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:11.954050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:11.954103 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:11.957798 (+ 3695us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:30:11.958463 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:11.959444 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:11.960211 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:11.960659 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:11.960788 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:11.964069 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:30:11.964075 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:30:11.964076 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:30:11.964093 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:30:11.966302 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:11.966691 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:11.966694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:11.966695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:11.966806 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:11.967015 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:11.967016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:11.967017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:11.967248 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:30:11.967400 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 09:30:11.967520 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 09:30:11.967683 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1024,"thread_start_us":74,"threads_started":1}
I20260926 09:30:11.975244 15789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386: Generated new TSK 0
I20260926 09:30:11.975474 15789 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:30:12.008759 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39239
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:30:12.091962 15803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:30:12.092190 15803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:30:12.092222 15803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:30:12.093552 15803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:30:12.093624 15803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:30:12.093654 15803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:30:12.095330 15803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:30:12.095608 15803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:30:12.095920 15803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:30:12.101212 15811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:30:12.101429 15803 server_base.cc:1034] running on GCE node
W20260926 09:30:12.101274 15809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:30:12.101214 15808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:30:12.101969 15803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:30:12.102268 15803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:30:12.103415 15803 hybrid_clock.cc:648] HybridClock initialized: now 1790415012103399 us; error 27 us; skew 500 ppm
Sep 26 09:30:12 dist-test-slave-dfzm krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:30:12.106180 15803 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:30:12.106664 15803 webserver.cc:460] Webserver started at http://127.0.98.1:45219/ using document root <none> and password file <none>
I20260926 09:30:12.106856 15803 fs_manager.cc:359] Metadata directory not provided
I20260926 09:30:12.106897 15803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:30:12.106985 15803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:30:12.107993 15803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "43448beb01ec4db1bc08f07e25d7f141"
format_stamp: "Formatted at 2026-09-26 09:30:12 on dist-test-slave-dfzm"
server_key: "392b5ace5dbd70617fc590a7dd3f638f"
server_key_iv: "d2e72e64c215f59b49b604063f94c12b"
server_key_version: "encryptionkey@0"
I20260926 09:30:12.108292 15803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "43448beb01ec4db1bc08f07e25d7f141"
format_stamp: "Formatted at 2026-09-26 09:30:12 on dist-test-slave-dfzm"
server_key: "392b5ace5dbd70617fc590a7dd3f638f"
server_key_iv: "d2e72e64c215f59b49b604063f94c12b"
server_key_version: "encryptionkey@0"
I20260926 09:30:12.109675 15803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:30:12.110569 15818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:12.110777 15803 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:30:12.110899 15803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "43448beb01ec4db1bc08f07e25d7f141"
format_stamp: "Formatted at 2026-09-26 09:30:12 on dist-test-slave-dfzm"
server_key: "392b5ace5dbd70617fc590a7dd3f638f"
server_key_iv: "d2e72e64c215f59b49b604063f94c12b"
server_key_version: "encryptionkey@0"
I20260926 09:30:12.110982 15803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:30:12.128836 15803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:30:12.131388 15803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:30:12.131558 15803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:30:12.131795 15803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:30:12.132125 15803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:30:12.132180 15803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:12.132223 15803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:30:12.132251 15803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:12.138012 15803 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43595
I20260926 09:30:12.138087 15931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43595 every 8 connection(s)
I20260926 09:30:12.138423 15803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:30:12.143886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 15803
I20260926 09:30:12.144021 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:30:12.144236 392 external_mini_cluster.cc:1428] Setting key 130170e477975a4b55efba8df71549a5
Sep 26 09:30:12 dist-test-slave-dfzm krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:30:12.148514 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:12.139174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39381 (local address 127.0.98.62:39239)
0926 09:30:12.139232 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:12.139235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:12.139472 (+ 237us) server_negotiation.cc:407] Connection header received
0926 09:30:12.139947 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:12.139947 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:12.139957 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:12.140007 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:30:12.140940 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.141453 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.142503 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.142707 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.142767 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:12.145083 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:30:12.145085 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:30:12.145085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:30:12.145093 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:30:12.146948 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:12.147455 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:12.147456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:12.147456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:12.147513 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:12.147852 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:12.147853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:12.147853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:12.147974 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:30:12.148038 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 09:30:12.148372 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 09:30:12.148447 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 09:30:12.148841 15934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:12.139158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.1:39381)
0926 09:30:12.139342 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:12.139377 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:30:12.139852 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:30:12.140067 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:30:12.140070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:30:12.140638 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:30:12.140828 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:12.140833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.141540 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.141541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:12.142416 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:12.142418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.142783 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.142784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:12.142880 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:12.143320 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:30:12.143328 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:30:12.144964 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:30:12.147034 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:30:12.147041 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:30:12.147043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:30:12.147375 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:30:12.147584 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:30:12.147585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:30:12.147585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:30:12.147737 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:30:12.148053 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:30:12.148059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:30:12.148230 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 09:30:12.148345 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 09:30:12.148558 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":96,"threads_started":1}
I20260926 09:30:12.149340 15932 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39239
I20260926 09:30:12.149461 15932 heartbeater.cc:464] Registering TS with master...
I20260926 09:30:12.149695 15932 heartbeater.cc:511] Master 127.0.98.62:39239 requested a full tablet report, sending...
I20260926 09:30:12.150265 15741 ts_manager.cc:194] Registered new tserver with Master: 43448beb01ec4db1bc08f07e25d7f141 (127.0.98.1:43595)
I20260926 09:30:12.151142 15741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39381
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:30:12.173225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39239
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:30:12.256349 15937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:30:12.256572 15937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:30:12.256596 15937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:30:12.257827 15937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:30:12.257871 15937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:30:12.257899 15937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:30:12.259331 15937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:30:12.259531 15937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:30:12.259709 15937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:30:12.265056 15945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:30:12.265028 15943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:30:12.265036 15942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:30:12.265094 15937 server_base.cc:1034] running on GCE node
I20260926 09:30:12.265533 15937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:30:12.265817 15937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:30:12.266966 15937 hybrid_clock.cc:648] HybridClock initialized: now 1790415012266952 us; error 28 us; skew 500 ppm
Sep 26 09:30:12 dist-test-slave-dfzm krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:30:12.269462 15937 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:30:12.269964 15937 webserver.cc:460] Webserver started at http://127.0.98.2:41135/ using document root <none> and password file <none>
I20260926 09:30:12.270146 15937 fs_manager.cc:359] Metadata directory not provided
I20260926 09:30:12.270186 15937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:30:12.270277 15937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:30:12.271278 15937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "be0ba63ea67542388ff599a1f1ba7440"
format_stamp: "Formatted at 2026-09-26 09:30:12 on dist-test-slave-dfzm"
server_key: "7aaec03837210a8863dc13c0ea963cc7"
server_key_iv: "4ef220e8741c2615fe18340c03a8cdc8"
server_key_version: "encryptionkey@0"
I20260926 09:30:12.271633 15937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "be0ba63ea67542388ff599a1f1ba7440"
format_stamp: "Formatted at 2026-09-26 09:30:12 on dist-test-slave-dfzm"
server_key: "7aaec03837210a8863dc13c0ea963cc7"
server_key_iv: "4ef220e8741c2615fe18340c03a8cdc8"
server_key_version: "encryptionkey@0"
I20260926 09:30:12.272902 15937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:30:12.273532 15952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:12.273715 15937 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:30:12.273819 15937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "be0ba63ea67542388ff599a1f1ba7440"
format_stamp: "Formatted at 2026-09-26 09:30:12 on dist-test-slave-dfzm"
server_key: "7aaec03837210a8863dc13c0ea963cc7"
server_key_iv: "4ef220e8741c2615fe18340c03a8cdc8"
server_key_version: "encryptionkey@0"
I20260926 09:30:12.273890 15937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:30:12.285527 15937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:30:12.287863 15937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:30:12.288024 15937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:30:12.288264 15937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:30:12.288575 15937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:30:12.288625 15937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:12.288666 15937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:30:12.288692 15937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:12.294610 15937 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40633
I20260926 09:30:12.294655 16065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40633 every 8 connection(s)
I20260926 09:30:12.294975 15937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:30:12.298118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 15937
I20260926 09:30:12.298245 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:30:12.298456 392 external_mini_cluster.cc:1428] Setting key 5084ea121d0b20a249f639eac0bc16ed
Sep 26 09:30:12 dist-test-slave-dfzm krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:30:12.305038 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:12.295649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44967 (local address 127.0.98.62:39239)
0926 09:30:12.295708 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:12.295710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:12.295890 (+ 180us) server_negotiation.cc:407] Connection header received
0926 09:30:12.296413 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:12.296414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:12.296427 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:12.296482 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:30:12.297499 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.298097 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.299482 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.299842 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.299917 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:12.302223 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:30:12.302225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:30:12.302226 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:30:12.302234 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:30:12.303654 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:12.304081 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:12.304082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:12.304083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:12.304142 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:12.304386 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:12.304387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:12.304387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:12.304506 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:30:12.304580 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 09:30:12.304851 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 09:30:12.304938 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 09:30:12.305254 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:12.295638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.2:44967)
0926 09:30:12.295775 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:12.295811 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 09:30:12.296303 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:30:12.296549 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:30:12.296553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:30:12.297126 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:30:12.297380 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:12.297386 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.298212 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.298213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:12.299408 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:12.299410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.299989 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.299990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:12.300143 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:12.300679 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:30:12.300692 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:30:12.302125 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:30:12.303756 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:30:12.303764 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:30:12.303766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:30:12.303995 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:30:12.304205 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:30:12.304206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:30:12.304206 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:30:12.304320 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:30:12.304572 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:30:12.304578 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:30:12.304723 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 09:30:12.304837 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 09:30:12.305003 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
I20260926 09:30:12.305933 16066 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39239
I20260926 09:30:12.306061 16066 heartbeater.cc:464] Registering TS with master...
I20260926 09:30:12.306303 16066 heartbeater.cc:511] Master 127.0.98.62:39239 requested a full tablet report, sending...
I20260926 09:30:12.306717 15741 ts_manager.cc:194] Registered new tserver with Master: be0ba63ea67542388ff599a1f1ba7440 (127.0.98.2:40633)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 09:30:12.307102 15741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:30:12.325028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39239
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:30:12.408670 16071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:30:12.408887 16071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:30:12.408911 16071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:30:12.410153 16071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:30:12.410202 16071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:30:12.410229 16071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:30:12.411628 16071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:30:12.411824 16071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:30:12.412000 16071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:30:12.417186 16076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:30:12.417179 16079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:30:12.417179 16077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:30:12.417505 16071 server_base.cc:1034] running on GCE node
I20260926 09:30:12.417742 16071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:30:12.418015 16071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:30:12.419157 16071 hybrid_clock.cc:648] HybridClock initialized: now 1790415012419144 us; error 29 us; skew 500 ppm
Sep 26 09:30:12 dist-test-slave-dfzm krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:30:12.421691 16071 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:30:12.422201 16071 webserver.cc:460] Webserver started at http://127.0.98.3:43367/ using document root <none> and password file <none>
I20260926 09:30:12.422461 16071 fs_manager.cc:359] Metadata directory not provided
I20260926 09:30:12.422530 16071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:30:12.422676 16071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:30:12.423736 16071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb"
format_stamp: "Formatted at 2026-09-26 09:30:12 on dist-test-slave-dfzm"
server_key: "826c3bf03016a63fb5ded37999d2c9f5"
server_key_iv: "88550bac1a3eb7b863d7d41481934fc2"
server_key_version: "encryptionkey@0"
I20260926 09:30:12.424094 16071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb"
format_stamp: "Formatted at 2026-09-26 09:30:12 on dist-test-slave-dfzm"
server_key: "826c3bf03016a63fb5ded37999d2c9f5"
server_key_iv: "88550bac1a3eb7b863d7d41481934fc2"
server_key_version: "encryptionkey@0"
I20260926 09:30:12.425482 16071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:30:12.426291 16086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:12.426479 16071 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:30:12.426582 16071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb"
format_stamp: "Formatted at 2026-09-26 09:30:12 on dist-test-slave-dfzm"
server_key: "826c3bf03016a63fb5ded37999d2c9f5"
server_key_iv: "88550bac1a3eb7b863d7d41481934fc2"
server_key_version: "encryptionkey@0"
I20260926 09:30:12.426656 16071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:30:12.452952 16071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:30:12.455148 16071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:30:12.455310 16071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:30:12.455543 16071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:30:12.455852 16071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:30:12.455904 16071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:12.455947 16071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:30:12.455972 16071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:12.461944 16071 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36181
I20260926 09:30:12.461987 16199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36181 every 8 connection(s)
I20260926 09:30:12.462333 16071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:30:12 dist-test-slave-dfzm krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:30:12.470075 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 16071
I20260926 09:30:12.470170 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:30:12.470394 392 external_mini_cluster.cc:1428] Setting key a84611da1a3c8c159ff4f953b3f8e3df
I20260926 09:30:12.472038 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:12.463071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54183 (local address 127.0.98.62:39239)
0926 09:30:12.463132 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:12.463135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:12.463346 (+ 211us) server_negotiation.cc:407] Connection header received
0926 09:30:12.463821 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:12.463822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:12.463831 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:12.463888 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:30:12.464863 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.465403 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.466527 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.466716 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.466772 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:12.468984 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:30:12.468986 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:30:12.468986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:30:12.468993 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:30:12.470650 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:12.471143 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:12.471144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:12.471144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:12.471196 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:12.471501 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:12.471502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:12.471502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:12.471613 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:30:12.471666 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 09:30:12.471909 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 09:30:12.471971 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 09:30:12.472206 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:12.463041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.3:54183)
0926 09:30:12.463221 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:12.463264 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:30:12.463740 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:30:12.463948 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:30:12.463951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:30:12.464550 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:30:12.464750 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:12.464754 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.465509 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.465510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:12.466447 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:12.466450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.466786 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.466787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:12.466897 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:12.467383 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:30:12.467391 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:30:12.468884 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:30:12.470746 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:30:12.470754 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:30:12.470757 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:30:12.471080 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:30:12.471279 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:30:12.471280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:30:12.471280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:30:12.471446 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:30:12.471673 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:30:12.471677 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:30:12.471790 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 09:30:12.471884 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 09:30:12.472007 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
I20260926 09:30:12.472627 16200 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39239
I20260926 09:30:12.472739 16200 heartbeater.cc:464] Registering TS with master...
I20260926 09:30:12.472916 16200 heartbeater.cc:511] Master 127.0.98.62:39239 requested a full tablet report, sending...
I20260926 09:30:12.473294 15741 ts_manager.cc:194] Registered new tserver with Master: daef8174a6bf4c75ab1f7fe0bbd7a9bb (127.0.98.3:36181)
I20260926 09:30:12.473728 15741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54183
I20260926 09:30:12.482276 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:30:12.490052 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:12.484053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53154 (local address 127.0.98.62:39239)
0926 09:30:12.484202 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:12.484204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:12.484350 (+ 146us) server_negotiation.cc:407] Connection header received
0926 09:30:12.484438 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:12.484439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:12.484451 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:12.484509 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:30:12.485035 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.485522 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.486271 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.486466 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.486521 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:12.487685 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:30:12.487686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:30:12.487687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:30:12.487692 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:30:12.488895 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:12.489264 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:12.489265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:12.489266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:12.489328 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:12.489506 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:12.489506 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:12.489506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:12.489714 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:30:12.489794 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 09:30:12.489881 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 09:30:12.489946 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260926 09:30:12.491480 15741 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:30:12.492985 15741 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:30:12.499256 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:12.495540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40633 (local address 127.0.0.1:45546)
0926 09:30:12.495951 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:12.495963 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:30:12.496022 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:30:12.496221 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:30:12.496222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:30:12.496224 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:30:12.496277 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:12.496279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.497196 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.497197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:12.498225 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:12.498228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.498965 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.498966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:12.499044 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:12.499058 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 09:30:12.499129 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 09:30:12.499183 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":88,"threads_started":1}
I20260926 09:30:12.499269 16215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:12.495656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45546 (local address 127.0.98.2:40633)
0926 09:30:12.495846 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:12.495851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:12.496085 (+ 234us) server_negotiation.cc:407] Connection header received
0926 09:30:12.496126 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:12.496128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:12.496140 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:12.496207 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:30:12.496370 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.497100 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.498330 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.498867 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.498963 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:12.499028 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 09:30:12.499144 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 09:30:12.499183 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":93,"threads_started":1}
I20260926 09:30:12.499974 16000 tablet_service.cc:1467] Processing CreateTablet for tablet fc6fc7c73fdf49eb8317e0601f58e502 (DEFAULT_TABLE table=test-table [id=793c408142ae452b859273b8f04cf2e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:30:12.500039 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:12.495316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36181 (local address 127.0.0.1:57538)
0926 09:30:12.495689 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:12.495734 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 09:30:12.495816 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:30:12.496106 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:30:12.496109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:30:12.496111 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:30:12.496186 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:12.496188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.497643 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.497645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:12.498869 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:12.498872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.499793 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.499794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:12.499873 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:12.499877 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:30:12.499928 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 09:30:12.499971 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":50,"thread_start_us":67,"threads_started":1}
I20260926 09:30:12.500329 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:12.495450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57146 (local address 127.0.98.1:43595)
0926 09:30:12.495649 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:12.495655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:12.495993 (+ 338us) server_negotiation.cc:407] Connection header received
0926 09:30:12.496060 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:12.496062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:12.496079 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:12.496150 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:30:12.496353 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.497523 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.498977 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.499696 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.499746 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:12.499803 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 09:30:12.500175 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 09:30:12.500225 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":98,"threads_started":1}
I20260926 09:30:12.500363 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc6fc7c73fdf49eb8317e0601f58e502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:30:12.500430 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:12.495769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57538 (local address 127.0.98.3:36181)
0926 09:30:12.495930 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:12.495935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:12.495948 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:30:12.495992 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:12.495995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:12.496011 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:12.496101 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:30:12.496388 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.497477 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.498973 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.499696 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.499766 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:12.499813 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 09:30:12.500296 (+ 483us) server_negotiation.cc:299] Negotiation successful
0926 09:30:12.500341 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
I20260926 09:30:12.500665 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:12.495316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43595 (local address 127.0.0.1:57146)
0926 09:30:12.495909 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:12.495918 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:30:12.495979 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:30:12.496261 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:30:12.496262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:30:12.496263 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:30:12.496300 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:12.496301 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.497654 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.497654 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:12.498874 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:12.498875 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.499997 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.499998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:12.500072 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:12.500076 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:30:12.500197 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 09:30:12.500592 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":37,"spinlock_wait_cycles":768,"thread_start_us":89,"threads_started":1}
I20260926 09:30:12.501015 16134 tablet_service.cc:1467] Processing CreateTablet for tablet fc6fc7c73fdf49eb8317e0601f58e502 (DEFAULT_TABLE table=test-table [id=793c408142ae452b859273b8f04cf2e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:30:12.501329 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc6fc7c73fdf49eb8317e0601f58e502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:30:12.501317 15866 tablet_service.cc:1467] Processing CreateTablet for tablet fc6fc7c73fdf49eb8317e0601f58e502 (DEFAULT_TABLE table=test-table [id=793c408142ae452b859273b8f04cf2e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:30:12.501677 15866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc6fc7c73fdf49eb8317e0601f58e502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:30:12.503840 16217 tablet_bootstrap.cc:492] T fc6fc7c73fdf49eb8317e0601f58e502 P be0ba63ea67542388ff599a1f1ba7440: Bootstrap starting.
I20260926 09:30:12.504171 16218 tablet_bootstrap.cc:492] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141: Bootstrap starting.
I20260926 09:30:12.504727 16219 tablet_bootstrap.cc:492] T fc6fc7c73fdf49eb8317e0601f58e502 P daef8174a6bf4c75ab1f7fe0bbd7a9bb: Bootstrap starting.
I20260926 09:30:12.504861 16218 tablet_bootstrap.cc:654] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141: Neither blocks nor log segments found. Creating new log.
I20260926 09:30:12.505031 16217 tablet_bootstrap.cc:654] T fc6fc7c73fdf49eb8317e0601f58e502 P be0ba63ea67542388ff599a1f1ba7440: Neither blocks nor log segments found. Creating new log.
I20260926 09:30:12.505390 16219 tablet_bootstrap.cc:654] T fc6fc7c73fdf49eb8317e0601f58e502 P daef8174a6bf4c75ab1f7fe0bbd7a9bb: Neither blocks nor log segments found. Creating new log.
I20260926 09:30:12.505398 16218 log.cc:824] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141: Log is configured to *not* fsync() on all Append() calls
I20260926 09:30:12.505677 16217 log.cc:824] T fc6fc7c73fdf49eb8317e0601f58e502 P be0ba63ea67542388ff599a1f1ba7440: Log is configured to *not* fsync() on all Append() calls
I20260926 09:30:12.506001 16219 log.cc:824] T fc6fc7c73fdf49eb8317e0601f58e502 P daef8174a6bf4c75ab1f7fe0bbd7a9bb: Log is configured to *not* fsync() on all Append() calls
I20260926 09:30:12.506279 16218 tablet_bootstrap.cc:492] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141: No bootstrap required, opened a new log
I20260926 09:30:12.506371 16218 ts_tablet_manager.cc:1397] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 09:30:12.506464 16217 tablet_bootstrap.cc:492] T fc6fc7c73fdf49eb8317e0601f58e502 P be0ba63ea67542388ff599a1f1ba7440: No bootstrap required, opened a new log
I20260926 09:30:12.506536 16217 ts_tablet_manager.cc:1397] T fc6fc7c73fdf49eb8317e0601f58e502 P be0ba63ea67542388ff599a1f1ba7440: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260926 09:30:12.506754 16219 tablet_bootstrap.cc:492] T fc6fc7c73fdf49eb8317e0601f58e502 P daef8174a6bf4c75ab1f7fe0bbd7a9bb: No bootstrap required, opened a new log
I20260926 09:30:12.506819 16219 ts_tablet_manager.cc:1397] T fc6fc7c73fdf49eb8317e0601f58e502 P daef8174a6bf4c75ab1f7fe0bbd7a9bb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:30:12.507892 16218 raft_consensus.cc:348] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } }
I20260926 09:30:12.508075 16218 raft_consensus.cc:374] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:30:12.508119 16218 raft_consensus.cc:729] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43448beb01ec4db1bc08f07e25d7f141, State: Initialized, Role: FOLLOWER
I20260926 09:30:12.508244 16218 consensus_queue.cc:260] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } }
I20260926 09:30:12.508510 16218 ts_tablet_manager.cc:1428] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 09:30:12.508639 15932 heartbeater.cc:503] Master 127.0.98.62:39239 was elected leader, sending a full tablet report...
I20260926 09:30:12.508543 16217 raft_consensus.cc:348] T fc6fc7c73fdf49eb8317e0601f58e502 P be0ba63ea67542388ff599a1f1ba7440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } }
I20260926 09:30:12.508705 16217 raft_consensus.cc:374] T fc6fc7c73fdf49eb8317e0601f58e502 P be0ba63ea67542388ff599a1f1ba7440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:30:12.508740 16217 raft_consensus.cc:729] T fc6fc7c73fdf49eb8317e0601f58e502 P be0ba63ea67542388ff599a1f1ba7440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be0ba63ea67542388ff599a1f1ba7440, State: Initialized, Role: FOLLOWER
I20260926 09:30:12.508759 16219 raft_consensus.cc:348] T fc6fc7c73fdf49eb8317e0601f58e502 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } }
I20260926 09:30:12.508854 16217 consensus_queue.cc:260] T fc6fc7c73fdf49eb8317e0601f58e502 P be0ba63ea67542388ff599a1f1ba7440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } }
I20260926 09:30:12.508903 16219 raft_consensus.cc:374] T fc6fc7c73fdf49eb8317e0601f58e502 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:30:12.508935 16219 raft_consensus.cc:729] T fc6fc7c73fdf49eb8317e0601f58e502 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daef8174a6bf4c75ab1f7fe0bbd7a9bb, State: Initialized, Role: FOLLOWER
I20260926 09:30:12.509081 16219 consensus_queue.cc:260] T fc6fc7c73fdf49eb8317e0601f58e502 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } }
I20260926 09:30:12.509131 16066 heartbeater.cc:503] Master 127.0.98.62:39239 was elected leader, sending a full tablet report...
I20260926 09:30:12.509142 16217 ts_tablet_manager.cc:1428] T fc6fc7c73fdf49eb8317e0601f58e502 P be0ba63ea67542388ff599a1f1ba7440: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 09:30:12.509348 16219 ts_tablet_manager.cc:1428] T fc6fc7c73fdf49eb8317e0601f58e502 P daef8174a6bf4c75ab1f7fe0bbd7a9bb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 09:30:12.509516 16200 heartbeater.cc:503] Master 127.0.98.62:39239 was elected leader, sending a full tablet report...
W20260926 09:30:12.545611 16067 tablet.cc:2369] T fc6fc7c73fdf49eb8317e0601f58e502 P be0ba63ea67542388ff599a1f1ba7440: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:30:12.553849 16223 raft_consensus.cc:482] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:30:12.554019 16223 raft_consensus.cc:504] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } }
I20260926 09:30:12.554451 16223 leader_election.cc:288] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers daef8174a6bf4c75ab1f7fe0bbd7a9bb (127.0.98.3:36181), be0ba63ea67542388ff599a1f1ba7440 (127.0.98.2:40633)
I20260926 09:30:12.557179 15934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:12.554633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36181 (local address 127.0.98.1:42473)
0926 09:30:12.554690 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:12.554703 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:30:12.554743 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:30:12.554910 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:30:12.554911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:30:12.554913 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:30:12.554986 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:12.554988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.555594 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.555596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:12.556365 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:12.556367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.556918 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.556919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:12.557001 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:12.557003 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:30:12.557061 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 09:30:12.557104 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 09:30:12.557179 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:12.554713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42473 (local address 127.0.98.3:36181)
0926 09:30:12.554765 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:12.554767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:12.554775 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:30:12.554802 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:12.554803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:12.554816 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:12.554890 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:30:12.555053 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.555515 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.556459 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.556839 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.556920 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:12.556931 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:30:12.557079 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 09:30:12.557119 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 09:30:12.557301 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:12.554634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40633 (local address 127.0.98.1:41351)
0926 09:30:12.554842 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:12.554851 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:30:12.554932 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:30:12.555080 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:30:12.555081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:30:12.555082 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:30:12.555152 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:12.555153 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.555766 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.555766 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:12.556474 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:12.556476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.557037 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.557038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:12.557115 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:12.557123 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 09:30:12.557187 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 09:30:12.557223 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":13,"thread_start_us":117,"threads_started":1}
I20260926 09:30:12.557294 16215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:12.554733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41351 (local address 127.0.98.2:40633)
0926 09:30:12.554782 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:12.554785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:12.554921 (+ 136us) server_negotiation.cc:407] Connection header received
0926 09:30:12.555003 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:12.555004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:12.555019 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:12.555077 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:30:12.555225 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.555674 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.556570 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.556939 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.557030 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:12.557039 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:30:12.557193 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 09:30:12.557233 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:30:12.557556 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6fc7c73fdf49eb8317e0601f58e502" candidate_uuid: "43448beb01ec4db1bc08f07e25d7f141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" is_pre_election: true
I20260926 09:30:12.557687 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6fc7c73fdf49eb8317e0601f58e502" candidate_uuid: "43448beb01ec4db1bc08f07e25d7f141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be0ba63ea67542388ff599a1f1ba7440" is_pre_election: true
I20260926 09:30:12.557753 16154 raft_consensus.cc:2454] T fc6fc7c73fdf49eb8317e0601f58e502 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43448beb01ec4db1bc08f07e25d7f141 in term 0.
I20260926 09:30:12.557806 16020 raft_consensus.cc:2454] T fc6fc7c73fdf49eb8317e0601f58e502 P be0ba63ea67542388ff599a1f1ba7440 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43448beb01ec4db1bc08f07e25d7f141 in term 0.
I20260926 09:30:12.557982 15821 leader_election.cc:302] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43448beb01ec4db1bc08f07e25d7f141, daef8174a6bf4c75ab1f7fe0bbd7a9bb; no voters:
I20260926 09:30:12.558126 16223 raft_consensus.cc:2781] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:30:12.558180 16223 raft_consensus.cc:482] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:30:12.558203 16223 raft_consensus.cc:3037] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:30:12.558885 16223 raft_consensus.cc:504] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } }
I20260926 09:30:12.559026 16223 leader_election.cc:288] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 [CANDIDATE]: Term 1 election: Requested vote from peers daef8174a6bf4c75ab1f7fe0bbd7a9bb (127.0.98.3:36181), be0ba63ea67542388ff599a1f1ba7440 (127.0.98.2:40633)
I20260926 09:30:12.559195 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6fc7c73fdf49eb8317e0601f58e502" candidate_uuid: "43448beb01ec4db1bc08f07e25d7f141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb"
I20260926 09:30:12.559202 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6fc7c73fdf49eb8317e0601f58e502" candidate_uuid: "43448beb01ec4db1bc08f07e25d7f141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be0ba63ea67542388ff599a1f1ba7440"
I20260926 09:30:12.559279 16020 raft_consensus.cc:3037] T fc6fc7c73fdf49eb8317e0601f58e502 P be0ba63ea67542388ff599a1f1ba7440 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:30:12.559279 16154 raft_consensus.cc:3037] T fc6fc7c73fdf49eb8317e0601f58e502 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:30:12.560153 16020 raft_consensus.cc:2454] T fc6fc7c73fdf49eb8317e0601f58e502 P be0ba63ea67542388ff599a1f1ba7440 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43448beb01ec4db1bc08f07e25d7f141 in term 1.
I20260926 09:30:12.560153 16154 raft_consensus.cc:2454] T fc6fc7c73fdf49eb8317e0601f58e502 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43448beb01ec4db1bc08f07e25d7f141 in term 1.
I20260926 09:30:12.560377 15820 leader_election.cc:302] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43448beb01ec4db1bc08f07e25d7f141, be0ba63ea67542388ff599a1f1ba7440; no voters:
I20260926 09:30:12.560496 16223 raft_consensus.cc:2781] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:30:12.560657 16223 raft_consensus.cc:686] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 [term 1 LEADER]: Becoming Leader. State: Replica: 43448beb01ec4db1bc08f07e25d7f141, State: Running, Role: LEADER
I20260926 09:30:12.560758 16223 consensus_queue.cc:237] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } }
I20260926 09:30:12.561571 15740 catalog_manager.cc:5495] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43448beb01ec4db1bc08f07e25d7f141 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "43448beb01ec4db1bc08f07e25d7f141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } health_report { overall_health: UNKNOWN } } }
I20260926 09:30:12.573256 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:12.570862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57148 (local address 127.0.98.1:43595)
0926 09:30:12.570901 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:12.570903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:12.570912 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:30:12.571003 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:12.571004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:12.571022 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:12.571082 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:30:12.571205 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.571706 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.572517 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:12.572694 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:12.572746 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:12.572936 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:30:12.573033 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:30:12.573094 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 09:30:12.573184 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 09:30:12.573201 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:30:12.575035 16020 raft_consensus.cc:1261] T fc6fc7c73fdf49eb8317e0601f58e502 P be0ba63ea67542388ff599a1f1ba7440 [term 1 FOLLOWER]: Refusing update from remote peer 43448beb01ec4db1bc08f07e25d7f141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:30:12.575042 16154 raft_consensus.cc:1261] T fc6fc7c73fdf49eb8317e0601f58e502 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [term 1 FOLLOWER]: Refusing update from remote peer 43448beb01ec4db1bc08f07e25d7f141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:30:12.575335 16223 consensus_queue.cc:1035] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:30:12.575456 16227 consensus_queue.cc:1035] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:30:12.577528 16230 mvcc.cc:205] Tried to move back new op lower bound from 7333539891504599040 to 7333539891449245696. Current Snapshot: MvccSnapshot[applied={T|T < 7333539891504599040}]
I20260926 09:30:12.577908 16232 mvcc.cc:205] Tried to move back new op lower bound from 7333539891504599040 to 7333539891449245696. Current Snapshot: MvccSnapshot[applied={T|T < 7333539891504599040}]
I20260926 09:30:12.580291 15741 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:30:12.580394 15741 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:30:12.581079 15741 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386: Sending DeleteTablet for 3 replicas of tablet fc6fc7c73fdf49eb8317e0601f58e502
I20260926 09:30:12.581372 16134 tablet_service.cc:1514] Processing DeleteTablet for tablet fc6fc7c73fdf49eb8317e0601f58e502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:30:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57538
I20260926 09:30:12.581506 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 15803
I20260926 09:30:12.581478 16000 tablet_service.cc:1514] Processing DeleteTablet for tablet fc6fc7c73fdf49eb8317e0601f58e502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:30:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45546
I20260926 09:30:12.581725 16240 tablet_replica.cc:330] T fc6fc7c73fdf49eb8317e0601f58e502 P be0ba63ea67542388ff599a1f1ba7440: stopping tablet replica
I20260926 09:30:12.581820 16240 raft_consensus.cc:2229] T fc6fc7c73fdf49eb8317e0601f58e502 P be0ba63ea67542388ff599a1f1ba7440 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:30:12.581910 16240 raft_consensus.cc:2258] T fc6fc7c73fdf49eb8317e0601f58e502 P be0ba63ea67542388ff599a1f1ba7440 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:30:12.582921 16239 tablet_replica.cc:330] T fc6fc7c73fdf49eb8317e0601f58e502 P daef8174a6bf4c75ab1f7fe0bbd7a9bb: stopping tablet replica
I20260926 09:30:12.582978 16240 ts_tablet_manager.cc:1905] T fc6fc7c73fdf49eb8317e0601f58e502 P be0ba63ea67542388ff599a1f1ba7440: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:30:12.583016 16239 raft_consensus.cc:2229] T fc6fc7c73fdf49eb8317e0601f58e502 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:30:12.583110 16239 raft_consensus.cc:2258] T fc6fc7c73fdf49eb8317e0601f58e502 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:30:12.583540 16239 ts_tablet_manager.cc:1905] T fc6fc7c73fdf49eb8317e0601f58e502 P daef8174a6bf4c75ab1f7fe0bbd7a9bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:30:12.584760 16240 ts_tablet_manager.cc:1918] T fc6fc7c73fdf49eb8317e0601f58e502 P be0ba63ea67542388ff599a1f1ba7440: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:30:12.584832 16240 log.cc:1196] T fc6fc7c73fdf49eb8317e0601f58e502 P be0ba63ea67542388ff599a1f1ba7440: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/wal/wals/fc6fc7c73fdf49eb8317e0601f58e502
I20260926 09:30:12.585132 16240 ts_tablet_manager.cc:1939] T fc6fc7c73fdf49eb8317e0601f58e502 P be0ba63ea67542388ff599a1f1ba7440: Deleting consensus metadata
I20260926 09:30:12.585176 16239 ts_tablet_manager.cc:1918] T fc6fc7c73fdf49eb8317e0601f58e502 P daef8174a6bf4c75ab1f7fe0bbd7a9bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:30:12.585240 16239 log.cc:1196] T fc6fc7c73fdf49eb8317e0601f58e502 P daef8174a6bf4c75ab1f7fe0bbd7a9bb: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/fc6fc7c73fdf49eb8317e0601f58e502
I20260926 09:30:12.585507 16239 ts_tablet_manager.cc:1939] T fc6fc7c73fdf49eb8317e0601f58e502 P daef8174a6bf4c75ab1f7fe0bbd7a9bb: Deleting consensus metadata
I20260926 09:30:12.585564 15726 catalog_manager.cc:4841] TS be0ba63ea67542388ff599a1f1ba7440 (127.0.98.2:40633): tablet fc6fc7c73fdf49eb8317e0601f58e502 (table test-table [id=793c408142ae452b859273b8f04cf2e5]) successfully deleted
I20260926 09:30:12.585971 15727 catalog_manager.cc:4841] TS daef8174a6bf4c75ab1f7fe0bbd7a9bb (127.0.98.3:36181): tablet fc6fc7c73fdf49eb8317e0601f58e502 (table test-table [id=793c408142ae452b859273b8f04cf2e5]) successfully deleted
W20260926 09:30:12.587731 15726 connection.cc:535] client connection to 127.0.98.1:43595 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 09:30:12.587827 15726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 09:30:12.588039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 15937
I20260926 09:30:12.588677 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:12.588215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43595 (local address 127.0.0.1:57164)
0926 09:30:12.588595 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:12.588636 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359}
W20260926 09:30:12.588920 15726 catalog_manager.cc:4584] TS 43448beb01ec4db1bc08f07e25d7f141 (127.0.98.1:43595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc6fc7c73fdf49eb8317e0601f58e502: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43595: connect: Connection refused (error 111)
I20260926 09:30:12.592701 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 16071
I20260926 09:30:12.597476 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 15709
I20260926 09:30:32.602659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39239
--webserver_interface=127.0.98.62
--webserver_port=42631
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:30:32.684006 16242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:30:32.684226 16242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:30:32.684250 16242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:30:32.685438 16242 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 09:30:32.685478 16242 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 09:30:32.685492 16242 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 09:30:32.685504 16242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:30:32.685552 16242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:30:32.685571 16242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:30:32.685622 16242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:30:32.685649 16242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:30:32.687172 16242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:30:32.687434 16242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:30:32.687667 16242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:30:32.696967 16249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:30:32.697009 16242 server_base.cc:1034] running on GCE node
W20260926 09:30:32.696908 16248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:30:32.696931 16251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:30:32.697515 16242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:30:32.697916 16242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:30:32.699069 16242 hybrid_clock.cc:648] HybridClock initialized: now 1790415032699041 us; error 42 us; skew 500 ppm
Sep 26 09:30:32 dist-test-slave-dfzm krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:30:32.701766 16242 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:30:32.702302 16242 webserver.cc:460] Webserver started at http://127.0.98.62:42631/ using document root <none> and password file <none>
I20260926 09:30:32.702548 16242 fs_manager.cc:359] Metadata directory not provided
I20260926 09:30:32.702634 16242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:30:32.704044 16242 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:30:32.704725 16258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:32.704875 16242 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:30:32.704985 16242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "9a9e0464302f4d018adea0b9aa6ca386"
format_stamp: "Formatted at 2026-09-26 09:30:11 on dist-test-slave-dfzm"
server_key: "00a89eeef900c5a3a023b164d153b862"
server_key_iv: "7e73e20109a3ec5079d2545bc86d59a6"
server_key_version: "encryptionkey@0"
I20260926 09:30:32.705407 16242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:30:32.726267 16242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:30:32.728387 16242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:30:32.728561 16242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:30:32.732928 16242 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39239
I20260926 09:30:32.732985 16310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39239 every 8 connection(s)
I20260926 09:30:32.733414 16242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:30:32.734965 16311 sys_catalog.cc:263] Verifying existing consensus state
I20260926 09:30:32.735575 16311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386: Bootstrap starting.
I20260926 09:30:32.737761 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 16242
I20260926 09:30:32.738106 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43595
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39239
--webserver_port=45219
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:30:32.744223 16311 log.cc:824] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386: Log is configured to *not* fsync() on all Append() calls
I20260926 09:30:32.746330 16311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:30:32.746639 16311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386: Bootstrap complete.
I20260926 09:30:32.748955 16311 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e0464302f4d018adea0b9aa6ca386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39239 } }
I20260926 09:30:32.749215 16311 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9e0464302f4d018adea0b9aa6ca386, State: Initialized, Role: FOLLOWER
I20260926 09:30:32.749400 16311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e0464302f4d018adea0b9aa6ca386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39239 } }
I20260926 09:30:32.749481 16311 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:30:32.749538 16311 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:30:32.749609 16311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 1 FOLLOWER]: Advancing to term 2
I20260926 09:30:32.750631 16311 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e0464302f4d018adea0b9aa6ca386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39239 } }
I20260926 09:30:32.750754 16311 leader_election.cc:302] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a9e0464302f4d018adea0b9aa6ca386; no voters:
I20260926 09:30:32.750980 16311 leader_election.cc:288] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 09:30:32.751019 16315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 09:30:32.751132 16315 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 2 LEADER]: Becoming Leader. State: Replica: 9a9e0464302f4d018adea0b9aa6ca386, State: Running, Role: LEADER
I20260926 09:30:32.751255 16311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:30:32.751283 16315 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e0464302f4d018adea0b9aa6ca386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39239 } }
I20260926 09:30:32.751577 16315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a9e0464302f4d018adea0b9aa6ca386. Latest consensus state: current_term: 2 leader_uuid: "9a9e0464302f4d018adea0b9aa6ca386" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e0464302f4d018adea0b9aa6ca386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39239 } } }
I20260926 09:30:32.751647 16315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [sys.catalog]: This master's current role is: LEADER
I20260926 09:30:32.751801 16316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9a9e0464302f4d018adea0b9aa6ca386" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e0464302f4d018adea0b9aa6ca386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39239 } } }
I20260926 09:30:32.751889 16316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [sys.catalog]: This master's current role is: LEADER
I20260926 09:30:32.752136 16319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:30:32.752945 16319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:30:32.753075 16319 catalog_manager.cc:1259] Loaded cluster ID: 56188d79318a40be977d4454fc5f2b8b
I20260926 09:30:32.753113 16319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:30:32.754261 16319 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:30:32.754460 16319 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386: Loaded TSK: 0
I20260926 09:30:32.768675 16319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386: Generated new TSK 1
I20260926 09:30:32.768800 16319 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 09:30:32.822665 16313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:30:32.822901 16313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:30:32.822942 16313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:30:32.824142 16313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:30:32.824210 16313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:30:32.824256 16313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:30:32.825902 16313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43595
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45219
--tserver_master_addrs=127.0.98.62:39239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:30:32.826169 16313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:30:32.826406 16313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:30:32.832404 16339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:30:32.832404 16341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:30:32.832612 16313 server_base.cc:1034] running on GCE node
W20260926 09:30:32.832726 16338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:30:32.832932 16313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:30:32.833216 16313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:30:32.834373 16313 hybrid_clock.cc:648] HybridClock initialized: now 1790415032834353 us; error 37 us; skew 500 ppm
Sep 26 09:30:32 dist-test-slave-dfzm krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:30:32.836861 16313 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:30:32.837364 16313 webserver.cc:460] Webserver started at http://127.0.98.1:45219/ using document root <none> and password file <none>
I20260926 09:30:32.837644 16313 fs_manager.cc:359] Metadata directory not provided
I20260926 09:30:32.837740 16313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:30:32.839156 16313 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:30:32.839926 16348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:32.840070 16313 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:30:32.840171 16313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "43448beb01ec4db1bc08f07e25d7f141"
format_stamp: "Formatted at 2026-09-26 09:30:12 on dist-test-slave-dfzm"
server_key: "392b5ace5dbd70617fc590a7dd3f638f"
server_key_iv: "d2e72e64c215f59b49b604063f94c12b"
server_key_version: "encryptionkey@0"
I20260926 09:30:32.840529 16313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:30:32.857875 16313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:30:32.860142 16313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:30:32.860283 16313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:30:32.860555 16313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:30:32.861073 16355 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 09:30:32.862584 16313 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 09:30:32.862633 16313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 09:30:32.862668 16313 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 09:30:32.863318 16313 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 09:30:32.863368 16313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 09:30:32.863412 16355 tablet_bootstrap.cc:492] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141: Bootstrap starting.
I20260926 09:30:32.871083 16313 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43595
I20260926 09:30:32.871627 16313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:30:32.872193 16462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43595 every 8 connection(s)
I20260926 09:30:32.873127 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 16313
I20260926 09:30:32.873483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40633
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39239
--webserver_port=41135
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:30:32.875082 16355 log.cc:824] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141: Log is configured to *not* fsync() on all Append() calls
I20260926 09:30:32.877568 16355 tablet_bootstrap.cc:492] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:30:32.877985 16355 tablet_bootstrap.cc:492] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141: Bootstrap complete.
I20260926 09:30:32.879194 16355 ts_tablet_manager.cc:1397] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.012s
Sep 26 09:30:32 dist-test-slave-dfzm krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:30:32.881035 16355 raft_consensus.cc:348] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } }
I20260926 09:30:32.881628 16355 raft_consensus.cc:729] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43448beb01ec4db1bc08f07e25d7f141, State: Initialized, Role: FOLLOWER
I20260926 09:30:32.881870 16355 consensus_queue.cc:260] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } }
I20260926 09:30:32.882262 16355 ts_tablet_manager.cc:1428] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 09:30:32.886269 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:32.872693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33861 (local address 127.0.98.62:39239)
0926 09:30:32.873917 (+ 1224us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:32.873926 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:32.873966 (+ 40us) server_negotiation.cc:407] Connection header received
0926 09:30:32.874095 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:32.874101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:32.874159 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:32.875095 (+ 936us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:30:32.875178 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:32.876594 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:32.878136 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:32.878764 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:32.881713 (+ 2949us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:32.881802 (+ 89us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:30:32.881805 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:30:32.881805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:30:32.881817 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:30:32.884175 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:32.884751 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:32.884754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:32.884755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:32.884859 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:32.885170 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:32.885171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:32.885171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:32.885446 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:30:32.885524 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 09:30:32.885937 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 09:30:32.886070 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1175,"thread_start_us":85,"threads_started":1}
I20260926 09:30:32.886634 16465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:32.872298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.1:33861)
0926 09:30:32.872739 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:32.872782 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:30:32.873451 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:30:32.874335 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:30:32.874339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:30:32.874703 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:30:32.874967 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:32.874974 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:32.876733 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:32.876734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:32.878037 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:32.878040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:32.878875 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:32.878876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:32.879019 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:32.879666 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:30:32.879676 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:30:32.881543 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:30:32.884268 (+ 2725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:30:32.884276 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:30:32.884278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:30:32.884648 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:30:32.884939 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:30:32.884940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:30:32.884941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:30:32.885095 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:30:32.885525 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:30:32.885532 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:30:32.885741 (+ 209us) client_negotiation.cc:769] Sending connection context
0926 09:30:32.885872 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 09:30:32.886071 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":82,"threads_started":1}
I20260926 09:30:32.887377 16463 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39239
I20260926 09:30:32.887480 16463 heartbeater.cc:464] Registering TS with master...
I20260926 09:30:32.887647 16463 heartbeater.cc:511] Master 127.0.98.62:39239 requested a full tablet report, sending...
I20260926 09:30:32.888365 16275 ts_manager.cc:194] Registered new tserver with Master: 43448beb01ec4db1bc08f07e25d7f141 (127.0.98.1:43595)
I20260926 09:30:32.889580 16275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33861
I20260926 09:30:32.890493 16463 heartbeater.cc:503] Master 127.0.98.62:39239 was elected leader, sending a full tablet report...
Sep 26 09:30:32 dist-test-slave-dfzm krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 09:30:32.898476 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:32.889009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43595 (local address 127.0.0.1:36190)
0926 09:30:32.889797 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:32.889834 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 09:30:32.889898 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:30:32.890189 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:30:32.890193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:30:32.890519 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:30:32.890638 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:32.890641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:32.891886 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:32.891887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:32.892844 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:32.892847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:32.893304 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:32.893307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:32.893400 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:32.893976 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:30:32.893982 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:30:32.895433 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:30:32.897193 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:30:32.897200 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:30:32.897203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:30:32.897411 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:30:32.897701 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:30:32.897702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:30:32.897702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:30:32.897767 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:30:32.898078 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:30:32.898084 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:30:32.898138 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 09:30:32.898232 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 09:30:32.898367 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"thread_start_us":84,"threads_started":1}
I20260926 09:30:32.898478 16471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:32.889224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36190 (local address 127.0.98.1:43595)
0926 09:30:32.890010 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:32.890014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:32.890022 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:30:32.890056 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:32.890058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:32.890071 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:32.890128 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:30:32.890724 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:32.891795 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:32.892950 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:32.893228 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:32.893290 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:32.895545 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:30:32.895555 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:30:32.895555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:30:32.895568 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:30:32.897076 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:32.897520 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:32.897524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:32.897524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:32.897627 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:32.897840 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:32.897841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:32.897841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:32.897994 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:30:32.898087 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 09:30:32.898285 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 09:30:32.898376 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":754,"thread_start_us":71,"threads_started":1}
I20260926 09:30:32.898921 16384 tablet_service.cc:1514] Processing DeleteTablet for tablet fc6fc7c73fdf49eb8317e0601f58e502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:30:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36190
I20260926 09:30:32.899523 16472 tablet_replica.cc:330] stopping tablet replica
I20260926 09:30:32.899641 16472 raft_consensus.cc:2229] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:30:32.899730 16472 raft_consensus.cc:2258] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:30:32.900023 16472 ts_tablet_manager.cc:1905] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:30:32.902233 16472 ts_tablet_manager.cc:1918] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:30:32.902321 16472 log.cc:1196] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/wal/wals/fc6fc7c73fdf49eb8317e0601f58e502
I20260926 09:30:32.902565 16472 ts_tablet_manager.cc:1939] T fc6fc7c73fdf49eb8317e0601f58e502 P 43448beb01ec4db1bc08f07e25d7f141: Deleting consensus metadata
I20260926 09:30:32.903054 16260 catalog_manager.cc:4841] TS 43448beb01ec4db1bc08f07e25d7f141 (127.0.98.1:43595): tablet fc6fc7c73fdf49eb8317e0601f58e502 (table test-table [id=793c408142ae452b859273b8f04cf2e5]) successfully deleted
W20260926 09:30:32.962376 16467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:30:32.962586 16467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:30:32.962611 16467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:30:32.963776 16467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:30:32.963819 16467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:30:32.963845 16467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:30:32.965329 16467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41135
--tserver_master_addrs=127.0.98.62:39239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:30:32.965574 16467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:30:32.965867 16467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:30:32.970831 16480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:30:32.970856 16477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:30:32.970921 16478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:30:32.971099 16467 server_base.cc:1034] running on GCE node
I20260926 09:30:32.971313 16467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:30:32.971596 16467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:30:32.972772 16467 hybrid_clock.cc:648] HybridClock initialized: now 1790415032972764 us; error 41 us; skew 500 ppm
Sep 26 09:30:32 dist-test-slave-dfzm krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:30:32.975229 16467 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:30:32.975687 16467 webserver.cc:460] Webserver started at http://127.0.98.2:41135/ using document root <none> and password file <none>
I20260926 09:30:32.975914 16467 fs_manager.cc:359] Metadata directory not provided
I20260926 09:30:32.975999 16467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:30:32.977387 16467 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:30:32.978121 16487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:32.978294 16467 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:30:32.978415 16467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "be0ba63ea67542388ff599a1f1ba7440"
format_stamp: "Formatted at 2026-09-26 09:30:12 on dist-test-slave-dfzm"
server_key: "7aaec03837210a8863dc13c0ea963cc7"
server_key_iv: "4ef220e8741c2615fe18340c03a8cdc8"
server_key_version: "encryptionkey@0"
I20260926 09:30:32.978765 16467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:30:32.985183 16467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:30:32.987085 16467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:30:32.987243 16467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:30:32.987484 16467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:30:32.987784 16467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:30:32.987838 16467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:32.987881 16467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:30:32.987903 16467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:32.993993 16467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40633
I20260926 09:30:32.994028 16600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40633 every 8 connection(s)
I20260926 09:30:32.994388 16467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:30:32 dist-test-slave-dfzm krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:30:33.000929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 16467
I20260926 09:30:33.001206 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36181
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39239
--webserver_port=43367
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:30:33.003975 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:32.995193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.2:58875)
0926 09:30:32.995351 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:32.995398 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 09:30:32.995860 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:30:32.996093 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:30:32.996097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:30:32.996657 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:30:32.996849 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:32.996854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:32.997513 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:32.997514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:32.998399 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:32.998401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:32.998745 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:32.998746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:32.998841 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:32.999400 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:30:32.999408 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:30:33.000758 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:30:33.002642 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:30:33.002648 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:30:33.002650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:30:33.002864 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:30:33.003078 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:30:33.003079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:30:33.003079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:30:33.003176 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:30:33.003471 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:30:33.003475 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:30:33.003594 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 09:30:33.003678 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 09:30:33.003803 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260926 09:30:33.003968 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:32.995180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58875 (local address 127.0.98.62:39239)
0926 09:30:32.995227 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:32.995230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:32.995485 (+ 255us) server_negotiation.cc:407] Connection header received
0926 09:30:32.995953 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:32.995955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:32.995967 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:32.996022 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:30:32.996931 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:32.997425 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:32.998473 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:32.998672 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:32.998726 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:33.000854 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:30:33.000855 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:30:33.000855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:30:33.000862 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:30:33.002556 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:33.002943 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:33.002944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:33.002945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:33.003011 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:33.003242 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:33.003243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:33.003243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:33.003405 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:30:33.003478 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 09:30:33.003726 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 09:30:33.003829 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:30:33.004501 16601 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39239
I20260926 09:30:33.004616 16601 heartbeater.cc:464] Registering TS with master...
I20260926 09:30:33.004827 16601 heartbeater.cc:511] Master 127.0.98.62:39239 requested a full tablet report, sending...
I20260926 09:30:33.005254 16275 ts_manager.cc:194] Registered new tserver with Master: be0ba63ea67542388ff599a1f1ba7440 (127.0.98.2:40633)
I20260926 09:30:33.005692 16275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58875
W20260926 09:30:33.084239 16604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:30:33.084501 16604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:30:33.084550 16604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:30:33.085876 16604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:30:33.085953 16604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:30:33.086004 16604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:30:33.087505 16604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43367
--tserver_master_addrs=127.0.98.62:39239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:30:33.087774 16604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:30:33.088014 16604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:30:33.093151 16612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:30:33.093158 16610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:30:33.093252 16609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:30:33.093329 16604 server_base.cc:1034] running on GCE node
I20260926 09:30:33.093703 16604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:30:33.093968 16604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:30:33.095137 16604 hybrid_clock.cc:648] HybridClock initialized: now 1790415033095121 us; error 38 us; skew 500 ppm
Sep 26 09:30:33 dist-test-slave-dfzm krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:30:33.097817 16604 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:30:33.098304 16604 webserver.cc:460] Webserver started at http://127.0.98.3:43367/ using document root <none> and password file <none>
I20260926 09:30:33.098555 16604 fs_manager.cc:359] Metadata directory not provided
I20260926 09:30:33.098639 16604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:30:33.100018 16604 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:30:33.100585 16619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:33.100742 16604 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:33.100847 16604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb"
format_stamp: "Formatted at 2026-09-26 09:30:12 on dist-test-slave-dfzm"
server_key: "826c3bf03016a63fb5ded37999d2c9f5"
server_key_iv: "88550bac1a3eb7b863d7d41481934fc2"
server_key_version: "encryptionkey@0"
I20260926 09:30:33.101190 16604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:30:33.110963 16604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:30:33.112874 16604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:30:33.113049 16604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:30:33.113301 16604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:30:33.113698 16604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:30:33.113750 16604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:33.113773 16604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:30:33.113786 16604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:33.119663 16604 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36181
I20260926 09:30:33.119741 16732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36181 every 8 connection(s)
I20260926 09:30:33.120085 16604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:30:33 dist-test-slave-dfzm krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:30:33.126806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 16604
I20260926 09:30:33.129908 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:33.120850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58749 (local address 127.0.98.62:39239)
0926 09:30:33.120915 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:33.120917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:33.121146 (+ 229us) server_negotiation.cc:407] Connection header received
0926 09:30:33.121688 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:33.121689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:33.121700 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:33.121748 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:30:33.122698 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.123187 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.124154 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.124335 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.124390 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:33.126624 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:30:33.126625 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:30:33.126626 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:30:33.126633 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:30:33.128353 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:33.128808 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:33.128809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:33.128810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:33.128870 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:33.129128 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:33.129130 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:33.129130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:33.129256 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:30:33.129338 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 09:30:33.129711 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 09:30:33.129799 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 09:30:33.130168 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:33.120834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.3:58749)
0926 09:30:33.121021 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:33.121063 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:30:33.121532 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:30:33.121813 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:30:33.121817 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:30:33.122418 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:30:33.122613 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:33.122619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.123270 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.123271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:33.124079 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:33.124081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.124418 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.124419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:33.124535 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:33.125035 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:30:33.125043 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:30:33.126524 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:30:33.128457 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:30:33.128465 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:30:33.128468 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:30:33.128708 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:30:33.128938 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:30:33.128939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:30:33.128940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:30:33.129059 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:30:33.129329 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:30:33.129335 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:30:33.129538 (+ 203us) client_negotiation.cc:769] Sending connection context
0926 09:30:33.129680 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 09:30:33.129879 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":91,"threads_started":1}
I20260926 09:30:33.130694 16733 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39239
I20260926 09:30:33.130788 16733 heartbeater.cc:464] Registering TS with master...
I20260926 09:30:33.130983 16733 heartbeater.cc:511] Master 127.0.98.62:39239 requested a full tablet report, sending...
I20260926 09:30:33.131361 16275 ts_manager.cc:194] Registered new tserver with Master: daef8174a6bf4c75ab1f7fe0bbd7a9bb (127.0.98.3:36181)
I20260926 09:30:33.131877 16275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58749
I20260926 09:30:33.134044 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:33.127229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37066 (local address 127.0.98.62:39239)
0926 09:30:33.127379 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:33.127381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:33.127519 (+ 138us) server_negotiation.cc:407] Connection header received
0926 09:30:33.127605 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:33.127605 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:33.127615 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:33.127685 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:30:33.128267 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.128750 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.129518 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.129855 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.129931 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:33.131413 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:30:33.131414 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:30:33.131415 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:30:33.131422 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:30:33.132948 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:33.133323 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:33.133324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:33.133324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:33.133384 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:33.133571 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:33.133572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:33.133572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:33.133740 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:30:33.133875 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 09:30:33.133907 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 09:30:33.133972 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":105,"threads_started":1}
I20260926 09:30:33.134760 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 09:30:33.137820 16205 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:39239)
I20260926 09:30:33.138558 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:33.135161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37072 (local address 127.0.98.62:39239)
0926 09:30:33.135203 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:33.135205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:33.135266 (+ 61us) server_negotiation.cc:407] Connection header received
0926 09:30:33.135358 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:33.135359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:33.135393 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:33.135484 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:30:33.135600 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.136059 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.136838 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.137048 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.137131 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:33.137312 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:30:33.137360 (+ 48us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 09:30:33.138473 (+ 1113us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37072: token expired
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 09:30:33.138620 16736 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37072: token expired
I20260926 09:30:33.145457 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:33.138416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37086 (local address 127.0.98.62:39239)
0926 09:30:33.138477 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:33.138478 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:33.138485 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:30:33.138501 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:33.138501 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:33.138509 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:33.138568 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:30:33.139116 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.139600 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.140672 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.140945 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.141016 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:33.142610 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:30:33.142611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:30:33.142612 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:30:33.142619 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:30:33.144373 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:33.144783 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:33.144784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:33.144784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:33.144858 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:33.145084 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:33.145085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:33.145085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:33.145182 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:30:33.145255 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 09:30:33.145342 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 09:30:33.145397 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:30:33.146596 16275 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:30:33.148207 16275 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:30:33.150658 16384 tablet_service.cc:1467] Processing CreateTablet for tablet b9f6262602354c22b032da9ab2e40866 (DEFAULT_TABLE table=test-table [id=5b485eab88d34122b30ee3d34828cbaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:30:33.150851 16384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f6262602354c22b032da9ab2e40866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:30:33.153265 16355 tablet_bootstrap.cc:492] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141: Bootstrap starting.
I20260926 09:30:33.153632 16355 tablet_bootstrap.cc:654] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141: Neither blocks nor log segments found. Creating new log.
I20260926 09:30:33.154153 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:33.150205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36181 (local address 127.0.0.1:39732)
0926 09:30:33.150281 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:33.150294 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:30:33.150337 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:30:33.150752 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:30:33.150753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:30:33.150754 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:30:33.150799 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:33.150801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.152188 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.152190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:33.153145 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:33.153147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.153883 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.153884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:33.153950 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:33.153979 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 09:30:33.154038 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 09:30:33.154085 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 09:30:33.154170 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:33.150428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39732 (local address 127.0.98.3:36181)
0926 09:30:33.150579 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:33.150583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:33.150593 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:30:33.150640 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:33.150641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:33.150653 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:33.150747 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:30:33.150859 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.152110 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.153239 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.153786 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.153882 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:33.153947 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 09:30:33.154054 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 09:30:33.154091 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260926 09:30:33.154299 16355 tablet_bootstrap.cc:492] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141: No bootstrap required, opened a new log
I20260926 09:30:33.154346 16355 ts_tablet_manager.cc:1397] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 09:30:33.154706 16355 raft_consensus.cc:348] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } }
I20260926 09:30:33.154760 16739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:33.150238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40633 (local address 127.0.0.1:55758)
0926 09:30:33.150383 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:33.150393 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:30:33.150438 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:30:33.150878 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:30:33.150879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:30:33.150881 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:30:33.150911 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:33.150913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.152175 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.152177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:33.153088 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:33.153090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.154511 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.154513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:33.154599 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:33.154603 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:30:33.154655 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 09:30:33.154691 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260926 09:30:33.154800 16355 raft_consensus.cc:374] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:30:33.154853 16355 raft_consensus.cc:729] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43448beb01ec4db1bc08f07e25d7f141, State: Initialized, Role: FOLLOWER
I20260926 09:30:33.154902 16355 consensus_queue.cc:260] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } }
I20260926 09:30:33.154861 16667 tablet_service.cc:1467] Processing CreateTablet for tablet b9f6262602354c22b032da9ab2e40866 (DEFAULT_TABLE table=test-table [id=5b485eab88d34122b30ee3d34828cbaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:30:33.154999 16355 ts_tablet_manager.cc:1428] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 09:30:33.155118 16740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:33.150329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55758 (local address 127.0.98.2:40633)
0926 09:30:33.150581 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:33.150586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:33.150595 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:30:33.150633 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:33.150635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:33.150650 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:33.150752 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:30:33.150998 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.152076 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.153173 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.154127 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.154197 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:33.154960 (+ 763us) server_negotiation.cc:1035] Waiting for connection context
0926 09:30:33.154988 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 09:30:33.155030 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":95,"threads_started":1}
I20260926 09:30:33.155210 16667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f6262602354c22b032da9ab2e40866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:30:33.155728 16535 tablet_service.cc:1467] Processing CreateTablet for tablet b9f6262602354c22b032da9ab2e40866 (DEFAULT_TABLE table=test-table [id=5b485eab88d34122b30ee3d34828cbaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:30:33.156044 16535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f6262602354c22b032da9ab2e40866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:30:33.158859 16743 tablet_bootstrap.cc:492] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb: Bootstrap starting.
I20260926 09:30:33.159483 16743 tablet_bootstrap.cc:654] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb: Neither blocks nor log segments found. Creating new log.
I20260926 09:30:33.159487 16744 tablet_bootstrap.cc:492] T b9f6262602354c22b032da9ab2e40866 P be0ba63ea67542388ff599a1f1ba7440: Bootstrap starting.
I20260926 09:30:33.160041 16744 tablet_bootstrap.cc:654] T b9f6262602354c22b032da9ab2e40866 P be0ba63ea67542388ff599a1f1ba7440: Neither blocks nor log segments found. Creating new log.
I20260926 09:30:33.160035 16743 log.cc:824] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb: Log is configured to *not* fsync() on all Append() calls
I20260926 09:30:33.160647 16744 log.cc:824] T b9f6262602354c22b032da9ab2e40866 P be0ba63ea67542388ff599a1f1ba7440: Log is configured to *not* fsync() on all Append() calls
I20260926 09:30:33.160969 16743 tablet_bootstrap.cc:492] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb: No bootstrap required, opened a new log
I20260926 09:30:33.161048 16743 ts_tablet_manager.cc:1397] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:30:33.161374 16744 tablet_bootstrap.cc:492] T b9f6262602354c22b032da9ab2e40866 P be0ba63ea67542388ff599a1f1ba7440: No bootstrap required, opened a new log
I20260926 09:30:33.161458 16744 ts_tablet_manager.cc:1397] T b9f6262602354c22b032da9ab2e40866 P be0ba63ea67542388ff599a1f1ba7440: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:30:33.162469 16743 raft_consensus.cc:348] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } }
I20260926 09:30:33.162606 16743 raft_consensus.cc:374] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:30:33.162628 16743 raft_consensus.cc:729] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daef8174a6bf4c75ab1f7fe0bbd7a9bb, State: Initialized, Role: FOLLOWER
I20260926 09:30:33.162739 16743 consensus_queue.cc:260] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } }
I20260926 09:30:33.163012 16743 ts_tablet_manager.cc:1428] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:30:33.163041 16733 heartbeater.cc:503] Master 127.0.98.62:39239 was elected leader, sending a full tablet report...
I20260926 09:30:33.162957 16744 raft_consensus.cc:348] T b9f6262602354c22b032da9ab2e40866 P be0ba63ea67542388ff599a1f1ba7440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } }
I20260926 09:30:33.163151 16744 raft_consensus.cc:374] T b9f6262602354c22b032da9ab2e40866 P be0ba63ea67542388ff599a1f1ba7440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:30:33.163192 16744 raft_consensus.cc:729] T b9f6262602354c22b032da9ab2e40866 P be0ba63ea67542388ff599a1f1ba7440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be0ba63ea67542388ff599a1f1ba7440, State: Initialized, Role: FOLLOWER
I20260926 09:30:33.163300 16744 consensus_queue.cc:260] T b9f6262602354c22b032da9ab2e40866 P be0ba63ea67542388ff599a1f1ba7440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } }
I20260926 09:30:33.163566 16744 ts_tablet_manager.cc:1428] T b9f6262602354c22b032da9ab2e40866 P be0ba63ea67542388ff599a1f1ba7440: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:30:33.163582 16601 heartbeater.cc:503] Master 127.0.98.62:39239 was elected leader, sending a full tablet report...
W20260926 09:30:33.245033 16602 tablet.cc:2369] T b9f6262602354c22b032da9ab2e40866 P be0ba63ea67542388ff599a1f1ba7440: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:30:33.313556 16747 raft_consensus.cc:482] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:30:33.313697 16747 raft_consensus.cc:504] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } }
I20260926 09:30:33.314128 16747 leader_election.cc:288] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be0ba63ea67542388ff599a1f1ba7440 (127.0.98.2:40633), 43448beb01ec4db1bc08f07e25d7f141 (127.0.98.1:43595)
I20260926 09:30:33.317294 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:33.314378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40633 (local address 127.0.98.3:39683)
0926 09:30:33.314459 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:33.314474 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 09:30:33.314546 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:30:33.314774 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:30:33.314775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:30:33.314778 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:30:33.314908 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:33.314911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.315617 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.315618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:33.316433 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:33.316436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.317015 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.317016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:33.317109 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:33.317114 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:30:33.317169 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 09:30:33.317212 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 09:30:33.317313 16740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:33.314455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39683 (local address 127.0.98.2:40633)
0926 09:30:33.314527 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:33.314530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:33.314543 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:30:33.314656 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:33.314658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:33.314671 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:33.314731 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:30:33.315028 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.315535 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.316537 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.316929 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.316999 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:33.317008 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:30:33.317193 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 09:30:33.317231 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 09:30:33.317459 16471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:33.314617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47149 (local address 127.0.98.1:43595)
0926 09:30:33.314667 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:33.314669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:33.314764 (+ 95us) server_negotiation.cc:407] Connection header received
0926 09:30:33.314811 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:33.314812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:33.314844 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:33.314926 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:30:33.315102 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.315589 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.316576 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.317098 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.317176 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:33.317259 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 09:30:33.317332 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 09:30:33.317381 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 09:30:33.317667 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:33.314509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43595 (local address 127.0.98.3:47149)
0926 09:30:33.314691 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:33.314700 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:30:33.314754 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:30:33.314933 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:30:33.314934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:30:33.314935 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:30:33.315024 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:33.315026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.315679 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.315679 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:33.316443 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:33.316445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.317198 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.317199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:33.317266 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:33.317270 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:30:33.317325 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 09:30:33.317565 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":5,"thread_start_us":103,"threads_started":1}
I20260926 09:30:33.317731 16555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f6262602354c22b032da9ab2e40866" candidate_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be0ba63ea67542388ff599a1f1ba7440" is_pre_election: true
I20260926 09:30:33.317970 16555 raft_consensus.cc:2454] T b9f6262602354c22b032da9ab2e40866 P be0ba63ea67542388ff599a1f1ba7440 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate daef8174a6bf4c75ab1f7fe0bbd7a9bb in term 0.
I20260926 09:30:33.317991 16417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f6262602354c22b032da9ab2e40866" candidate_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43448beb01ec4db1bc08f07e25d7f141" is_pre_election: true
I20260926 09:30:33.318167 16417 raft_consensus.cc:2454] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate daef8174a6bf4c75ab1f7fe0bbd7a9bb in term 0.
I20260926 09:30:33.318312 16621 leader_election.cc:302] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be0ba63ea67542388ff599a1f1ba7440, daef8174a6bf4c75ab1f7fe0bbd7a9bb; no voters:
I20260926 09:30:33.318442 16747 raft_consensus.cc:2781] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:30:33.318507 16747 raft_consensus.cc:482] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:30:33.318538 16747 raft_consensus.cc:3037] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:30:33.319262 16747 raft_consensus.cc:504] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } }
I20260926 09:30:33.319402 16747 leader_election.cc:288] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [CANDIDATE]: Term 1 election: Requested vote from peers be0ba63ea67542388ff599a1f1ba7440 (127.0.98.2:40633), 43448beb01ec4db1bc08f07e25d7f141 (127.0.98.1:43595)
I20260926 09:30:33.319600 16555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f6262602354c22b032da9ab2e40866" candidate_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be0ba63ea67542388ff599a1f1ba7440"
I20260926 09:30:33.319609 16417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f6262602354c22b032da9ab2e40866" candidate_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43448beb01ec4db1bc08f07e25d7f141"
I20260926 09:30:33.319680 16555 raft_consensus.cc:3037] T b9f6262602354c22b032da9ab2e40866 P be0ba63ea67542388ff599a1f1ba7440 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:30:33.319715 16417 raft_consensus.cc:3037] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:30:33.320441 16555 raft_consensus.cc:2454] T b9f6262602354c22b032da9ab2e40866 P be0ba63ea67542388ff599a1f1ba7440 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate daef8174a6bf4c75ab1f7fe0bbd7a9bb in term 1.
I20260926 09:30:33.320444 16417 raft_consensus.cc:2454] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate daef8174a6bf4c75ab1f7fe0bbd7a9bb in term 1.
I20260926 09:30:33.320614 16621 leader_election.cc:302] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be0ba63ea67542388ff599a1f1ba7440, daef8174a6bf4c75ab1f7fe0bbd7a9bb; no voters:
I20260926 09:30:33.320785 16747 raft_consensus.cc:2781] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:30:33.320925 16747 raft_consensus.cc:686] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [term 1 LEADER]: Becoming Leader. State: Replica: daef8174a6bf4c75ab1f7fe0bbd7a9bb, State: Running, Role: LEADER
I20260926 09:30:33.321031 16747 consensus_queue.cc:237] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } }
I20260926 09:30:33.321908 16274 catalog_manager.cc:5495] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb reported cstate change: term changed from 0 to 1, leader changed from <none> to daef8174a6bf4c75ab1f7fe0bbd7a9bb (127.0.98.3). New cstate: current_term: 1 leader_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } health_report { overall_health: UNKNOWN } } }
I20260926 09:30:33.337975 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:33.335536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39734 (local address 127.0.98.3:36181)
0926 09:30:33.335590 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:33.335592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:33.335602 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:30:33.335690 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:33.335691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:33.335711 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:33.335790 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:30:33.335899 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.336366 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.337111 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:33.337304 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:33.337382 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:33.337567 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:30:33.337718 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:30:33.337816 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 09:30:33.337841 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 09:30:33.337862 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 09:30:33.339825 16417 raft_consensus.cc:1261] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [term 1 FOLLOWER]: Refusing update from remote peer daef8174a6bf4c75ab1f7fe0bbd7a9bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:30:33.339907 16555 raft_consensus.cc:1261] T b9f6262602354c22b032da9ab2e40866 P be0ba63ea67542388ff599a1f1ba7440 [term 1 FOLLOWER]: Refusing update from remote peer daef8174a6bf4c75ab1f7fe0bbd7a9bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:30:33.340220 16747 consensus_queue.cc:1035] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:30:33.340355 16750 consensus_queue.cc:1035] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:30:33.343590 16755 mvcc.cc:205] Tried to move back new op lower bound from 7333539976557154304 to 7333539976483532800. Current Snapshot: MvccSnapshot[applied={T|T < 7333539976557154304}]
I20260926 09:30:33.345443 16275 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:30:33.345551 16275 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:30:33.346330 16275 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386: Sending DeleteTablet for 3 replicas of tablet b9f6262602354c22b032da9ab2e40866
I20260926 09:30:33.346719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 16313
I20260926 09:30:33.346699 16667 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f6262602354c22b032da9ab2e40866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:30:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39732
I20260926 09:30:33.346935 16535 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f6262602354c22b032da9ab2e40866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:30:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55758
I20260926 09:30:33.347415 16763 tablet_replica.cc:330] T b9f6262602354c22b032da9ab2e40866 P be0ba63ea67542388ff599a1f1ba7440: stopping tablet replica
I20260926 09:30:33.347540 16763 raft_consensus.cc:2229] T b9f6262602354c22b032da9ab2e40866 P be0ba63ea67542388ff599a1f1ba7440 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:30:33.347652 16763 raft_consensus.cc:2258] T b9f6262602354c22b032da9ab2e40866 P be0ba63ea67542388ff599a1f1ba7440 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:30:33.348107 16763 ts_tablet_manager.cc:1905] T b9f6262602354c22b032da9ab2e40866 P be0ba63ea67542388ff599a1f1ba7440: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:30:33.349884 16763 ts_tablet_manager.cc:1918] T b9f6262602354c22b032da9ab2e40866 P be0ba63ea67542388ff599a1f1ba7440: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:30:33.349960 16763 log.cc:1196] T b9f6262602354c22b032da9ab2e40866 P be0ba63ea67542388ff599a1f1ba7440: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/wal/wals/b9f6262602354c22b032da9ab2e40866
I20260926 09:30:33.350248 16763 ts_tablet_manager.cc:1939] T b9f6262602354c22b032da9ab2e40866 P be0ba63ea67542388ff599a1f1ba7440: Deleting consensus metadata
I20260926 09:30:33.350708 16260 catalog_manager.cc:4841] TS be0ba63ea67542388ff599a1f1ba7440 (127.0.98.2:40633): tablet b9f6262602354c22b032da9ab2e40866 (table test-table [id=5b485eab88d34122b30ee3d34828cbaa]) successfully deleted
W20260926 09:30:33.352921 16260 connection.cc:535] client connection to 127.0.98.1:43595 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 09:30:33.353029 16260 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 09:30:33.353135 16762 tablet_replica.cc:330] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb: stopping tablet replica
I20260926 09:30:33.353255 16762 raft_consensus.cc:2229] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:30:33.353371 16762 raft_consensus.cc:2258] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:30:33.353642 16739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:33.353476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43595 (local address 127.0.0.1:36198)
0926 09:30:33.353543 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:33.353601 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 09:30:33.353706 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 16467
I20260926 09:30:33.354465 16762 ts_tablet_manager.cc:1905] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:30:33.356195 16762 ts_tablet_manager.cc:1918] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:30:33.356259 16762 log.cc:1196] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/b9f6262602354c22b032da9ab2e40866
I20260926 09:30:33.356546 16762 ts_tablet_manager.cc:1939] T b9f6262602354c22b032da9ab2e40866 P daef8174a6bf4c75ab1f7fe0bbd7a9bb: Deleting consensus metadata
I20260926 09:30:33.357426 16261 catalog_manager.cc:4841] TS daef8174a6bf4c75ab1f7fe0bbd7a9bb (127.0.98.3:36181): tablet b9f6262602354c22b032da9ab2e40866 (table test-table [id=5b485eab88d34122b30ee3d34828cbaa]) successfully deleted
W20260926 09:30:33.358217 16260 catalog_manager.cc:4584] TS 43448beb01ec4db1bc08f07e25d7f141 (127.0.98.1:43595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b9f6262602354c22b032da9ab2e40866: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43595: connect: Connection refused (error 111)
I20260926 09:30:33.359705 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 16604
I20260926 09:30:33.364017 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 16242
I20260926 09:30:53.369709 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39239
--webserver_interface=127.0.98.62
--webserver_port=42631
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:30:53.463021 16765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:30:53.463395 16765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:30:53.463475 16765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:30:53.465047 16765 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 09:30:53.465103 16765 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 09:30:53.465117 16765 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 09:30:53.465134 16765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:30:53.465148 16765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:30:53.465245 16765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:30:53.465266 16765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:30:53.465292 16765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:30:53.467005 16765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:30:53.467320 16765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:30:53.467597 16765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:30:53.473145 16771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:30:53.473134 16770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:30:53.473134 16773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:30:53.473486 16765 server_base.cc:1034] running on GCE node
I20260926 09:30:53.473977 16765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:30:53.474255 16765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:30:53.475507 16765 hybrid_clock.cc:648] HybridClock initialized: now 1790415053475495 us; error 31 us; skew 500 ppm
Sep 26 09:30:53 dist-test-slave-dfzm krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:30:53.478495 16765 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:30:53.479007 16765 webserver.cc:460] Webserver started at http://127.0.98.62:42631/ using document root <none> and password file <none>
I20260926 09:30:53.479215 16765 fs_manager.cc:359] Metadata directory not provided
I20260926 09:30:53.479264 16765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:30:53.480587 16765 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:30:53.481359 16780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:53.481616 16765 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:30:53.481705 16765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "9a9e0464302f4d018adea0b9aa6ca386"
format_stamp: "Formatted at 2026-09-26 09:30:11 on dist-test-slave-dfzm"
server_key: "00a89eeef900c5a3a023b164d153b862"
server_key_iv: "7e73e20109a3ec5079d2545bc86d59a6"
server_key_version: "encryptionkey@0"
I20260926 09:30:53.482086 16765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:30:53.507073 16765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:30:53.509781 16765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:30:53.509919 16765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:30:53.513839 16765 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39239
I20260926 09:30:53.513932 16832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39239 every 8 connection(s)
I20260926 09:30:53.514421 16765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:30:53.515122 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 16765
I20260926 09:30:53.515460 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43595
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39239
--webserver_port=45219
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:30:53.516842 16833 sys_catalog.cc:263] Verifying existing consensus state
I20260926 09:30:53.517531 16833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386: Bootstrap starting.
I20260926 09:30:53.526067 16833 log.cc:824] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386: Log is configured to *not* fsync() on all Append() calls
I20260926 09:30:53.528716 16833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:30:53.529085 16833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386: Bootstrap complete.
I20260926 09:30:53.531462 16833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e0464302f4d018adea0b9aa6ca386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39239 } }
I20260926 09:30:53.531689 16833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9e0464302f4d018adea0b9aa6ca386, State: Initialized, Role: FOLLOWER
I20260926 09:30:53.531874 16833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e0464302f4d018adea0b9aa6ca386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39239 } }
I20260926 09:30:53.531944 16833 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:30:53.532016 16833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:30:53.532059 16833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 2 FOLLOWER]: Advancing to term 3
I20260926 09:30:53.533083 16833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e0464302f4d018adea0b9aa6ca386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39239 } }
I20260926 09:30:53.533172 16833 leader_election.cc:302] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a9e0464302f4d018adea0b9aa6ca386; no voters:
I20260926 09:30:53.533417 16833 leader_election.cc:288] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 09:30:53.533455 16837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 09:30:53.533617 16837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [term 3 LEADER]: Becoming Leader. State: Replica: 9a9e0464302f4d018adea0b9aa6ca386, State: Running, Role: LEADER
I20260926 09:30:53.533725 16833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:30:53.533711 16837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e0464302f4d018adea0b9aa6ca386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39239 } }
I20260926 09:30:53.534011 16837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a9e0464302f4d018adea0b9aa6ca386. Latest consensus state: current_term: 3 leader_uuid: "9a9e0464302f4d018adea0b9aa6ca386" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e0464302f4d018adea0b9aa6ca386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39239 } } }
I20260926 09:30:53.534085 16837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [sys.catalog]: This master's current role is: LEADER
I20260926 09:30:53.534466 16838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9a9e0464302f4d018adea0b9aa6ca386" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e0464302f4d018adea0b9aa6ca386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39239 } } }
I20260926 09:30:53.534621 16842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:30:53.534621 16838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386 [sys.catalog]: This master's current role is: LEADER
I20260926 09:30:53.535758 16842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:30:53.535882 16842 catalog_manager.cc:1259] Loaded cluster ID: 56188d79318a40be977d4454fc5f2b8b
I20260926 09:30:53.535928 16842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:30:53.536914 16842 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:30:53.537110 16842 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386: Loaded TSK: 0
I20260926 09:30:53.537150 16842 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9a9e0464302f4d018adea0b9aa6ca386: Loaded TSK: 1
F20260926 09:30:53.537372 16842 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe294fc99ad google::LogMessage::Fail() at ??:0
@ 0x7fe294fcc88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe294fc94de google::LogMessage::Flush() at ??:0
@ 0x7fe294fcaa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe297736104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe297733d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe29cf231cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe29cf2348e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe29cf23594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fe29cee9c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fe29cef3ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe296e970d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe296e8c954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe297cf06db start_thread at ??:0
@ 0x7fe293ac761f clone at ??:0
W20260926 09:30:53.620129 16835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:30:53.620352 16835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:30:53.620376 16835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:30:53.621640 16835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:30:53.621701 16835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:30:53.621735 16835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:30:53.623288 16835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43595
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45219
--tserver_master_addrs=127.0.98.62:39239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:30:53.623541 16835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:30:53.623834 16835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:30:53.629282 16874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:30:53.629302 16877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:30:53.629298 16875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:30:53.629638 16835 server_base.cc:1034] running on GCE node
I20260926 09:30:53.629912 16835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:30:53.630192 16835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:30:53.631436 16835 hybrid_clock.cc:648] HybridClock initialized: now 1790415053631400 us; error 34 us; skew 500 ppm
Sep 26 09:30:53 dist-test-slave-dfzm krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:30:53.634176 16835 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:30:53.634740 16835 webserver.cc:460] Webserver started at http://127.0.98.1:45219/ using document root <none> and password file <none>
I20260926 09:30:53.634989 16835 fs_manager.cc:359] Metadata directory not provided
I20260926 09:30:53.635062 16835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:30:53.636482 16835 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:30:53.637210 16884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:53.637418 16835 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:30:53.637531 16835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "43448beb01ec4db1bc08f07e25d7f141"
format_stamp: "Formatted at 2026-09-26 09:30:12 on dist-test-slave-dfzm"
server_key: "392b5ace5dbd70617fc590a7dd3f638f"
server_key_iv: "d2e72e64c215f59b49b604063f94c12b"
server_key_version: "encryptionkey@0"
I20260926 09:30:53.637923 16835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:30:53.647729 16835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:30:53.650118 16835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:30:53.650269 16835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:30:53.650525 16835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:30:53.650997 16891 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 09:30:53.652889 16835 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 09:30:53.652966 16835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 09:30:53.653010 16835 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 09:30:53.653699 16835 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 09:30:53.653757 16835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260926 09:30:53.653802 16891 tablet_bootstrap.cc:492] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141: Bootstrap starting.
I20260926 09:30:53.661239 16835 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43595
I20260926 09:30:53.661424 16998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43595 every 8 connection(s)
I20260926 09:30:53.661778 16835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:30:53.665272 16891 log.cc:824] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141: Log is configured to *not* fsync() on all Append() calls
I20260926 09:30:53.667536 16891 tablet_bootstrap.cc:492] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:30:53.667865 16891 tablet_bootstrap.cc:492] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141: Bootstrap complete.
I20260926 09:30:53.668797 16891 ts_tablet_manager.cc:1397] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20260926 09:30:53.669951 16891 raft_consensus.cc:348] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } }
I20260926 09:30:53.670212 16891 raft_consensus.cc:729] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43448beb01ec4db1bc08f07e25d7f141, State: Initialized, Role: FOLLOWER
I20260926 09:30:53.670403 16891 consensus_queue.cc:260] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } }
I20260926 09:30:53.670688 16891 ts_tablet_manager.cc:1428] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141: Time spent starting tablet: real 0.002s user 0.004s sys 0.001s
I20260926 09:30:53.671047 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 16835
I20260926 09:30:53.671319 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40633
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39239
--webserver_port=41135
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:30:53.736037 17001 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:30:53.662494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.1:57857)
0926 09:30:53.662679 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:53.662722 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:30:53.663192 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:30:53.735835 (+ 72643us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":113,"threads_started":1}
W20260926 09:30:53.736271 16999 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39239 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39239: Client connection negotiation failed: client connection to 127.0.98.62:39239: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 09:30:53.736629 17001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:53.736497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.1:36073)
0926 09:30:53.736560 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:53.736600 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 09:30:53.737018 17001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:53.736927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.1:53053)
0926 09:30:53.736979 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:53.737003 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 09:30:53.737161 16999 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 09:30:53.755339 17004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:30:53.755592 17004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:30:53.755635 17004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:30:53.756815 17004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:30:53.756891 17004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:30:53.756943 17004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:30:53.758509 17004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41135
--tserver_master_addrs=127.0.98.62:39239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:30:53.758786 17004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:30:53.759024 17004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:30:53.764227 17012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:30:53.764227 17009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:30:53.764287 17010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:30:53.764468 17004 server_base.cc:1034] running on GCE node
I20260926 09:30:53.764684 17004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:30:53.764945 17004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:30:53.766148 17004 hybrid_clock.cc:648] HybridClock initialized: now 1790415053766132 us; error 46 us; skew 500 ppm
Sep 26 09:30:53 dist-test-slave-dfzm krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:30:53.768836 17004 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:30:53.769371 17004 webserver.cc:460] Webserver started at http://127.0.98.2:41135/ using document root <none> and password file <none>
I20260926 09:30:53.769683 17004 fs_manager.cc:359] Metadata directory not provided
I20260926 09:30:53.769737 17004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:30:53.771166 17004 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:30:53.771977 17019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:53.772193 17004 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:30:53.772280 17004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "be0ba63ea67542388ff599a1f1ba7440"
format_stamp: "Formatted at 2026-09-26 09:30:12 on dist-test-slave-dfzm"
server_key: "7aaec03837210a8863dc13c0ea963cc7"
server_key_iv: "4ef220e8741c2615fe18340c03a8cdc8"
server_key_version: "encryptionkey@0"
I20260926 09:30:53.772655 17004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:30:53.787062 17004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:30:53.789669 17004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:30:53.789795 17004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:30:53.789994 17004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:30:53.790316 17004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:30:53.790354 17004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:53.790377 17004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:30:53.790436 17004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:53.796368 17004 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40633
I20260926 09:30:53.796609 17132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40633 every 8 connection(s)
I20260926 09:30:53.796870 17004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:30:53.798097 17135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:53.797723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.2:42159)
0926 09:30:53.797879 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:53.797958 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
W20260926 09:30:53.798557 17133 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39239 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39239: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
I20260926 09:30:53.798871 17135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:53.798776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.2:46779)
0926 09:30:53.798826 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:53.798854 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 09:30:53.799136 17135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:53.799089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.2:33573)
0926 09:30:53.799108 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:53.799125 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260926 09:30:53.799261 17133 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 09:30:53.806890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 17004
I20260926 09:30:53.807214 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36181
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39239
--webserver_port=43367
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:30:53.889761 17136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:30:53.889994 17136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:30:53.890033 17136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:30:53.891198 17136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:30:53.891281 17136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:30:53.891328 17136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:30:53.892805 17136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43367
--tserver_master_addrs=127.0.98.62:39239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:30:53.893074 17136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:30:53.893316 17136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:30:53.898579 17142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:30:53.898666 17141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:30:53.898581 17144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:30:53.898799 17136 server_base.cc:1034] running on GCE node
I20260926 09:30:53.899008 17136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:30:53.899278 17136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:30:53.900463 17136 hybrid_clock.cc:648] HybridClock initialized: now 1790415053900368 us; error 110 us; skew 500 ppm
Sep 26 09:30:53 dist-test-slave-dfzm krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:30:53.903203 17136 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:30:53.903687 17136 webserver.cc:460] Webserver started at http://127.0.98.3:43367/ using document root <none> and password file <none>
I20260926 09:30:53.903939 17136 fs_manager.cc:359] Metadata directory not provided
I20260926 09:30:53.904017 17136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:30:53.905427 17136 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:30:53.906193 17151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:53.906423 17136 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:30:53.906533 17136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb"
format_stamp: "Formatted at 2026-09-26 09:30:12 on dist-test-slave-dfzm"
server_key: "826c3bf03016a63fb5ded37999d2c9f5"
server_key_iv: "88550bac1a3eb7b863d7d41481934fc2"
server_key_version: "encryptionkey@0"
I20260926 09:30:53.906903 17136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:30:53.933703 17136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:30:53.935933 17136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:30:53.936142 17136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:30:53.936411 17136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:30:53.936738 17136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:30:53.936800 17136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:53.936842 17136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:30:53.936872 17136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:30:53.943159 17136 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36181
I20260926 09:30:53.943218 17264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36181 every 8 connection(s)
I20260926 09:30:53.943557 17136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:30:53.944666 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:53.944290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.3:41697)
0926 09:30:53.944451 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:53.944526 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
W20260926 09:30:53.945127 17265 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39239 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39239: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
I20260926 09:30:53.945501 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:53.945401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.3:39719)
0926 09:30:53.945453 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:53.945483 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 09:30:53.945854 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:53.945776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.3:43559)
0926 09:30:53.945818 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:53.945840 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 09:30:53.945989 17265 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 09:30:53.952669 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 17136
I20260926 09:30:54.737890 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:54.737561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.1:44969)
0926 09:30:54.737742 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:54.737783 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":117,"threads_started":1}
I20260926 09:30:54.799834 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:54.799604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.2:47291)
0926 09:30:54.799746 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:54.799787 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
I20260926 09:30:54.946707 17271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:54.946446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.3:44843)
0926 09:30:54.946612 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:54.946655 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":79,"threads_started":1}
I20260926 09:30:55.466410 17272 raft_consensus.cc:482] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:30:55.466537 17272 raft_consensus.cc:504] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } }
I20260926 09:30:55.466960 17272 leader_election.cc:288] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers daef8174a6bf4c75ab1f7fe0bbd7a9bb (127.0.98.3:36181), be0ba63ea67542388ff599a1f1ba7440 (127.0.98.2:40633)
Sep 26 09:30:55 dist-test-slave-dfzm krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 26 09:30:55 dist-test-slave-dfzm krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260926 09:30:55.481503 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:55.467393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53167 (local address 127.0.98.3:36181)
0926 09:30:55.467555 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:55.467564 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:55.467583 (+ 19us) server_negotiation.cc:407] Connection header received
0926 09:30:55.468018 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:55.468021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:55.468062 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:55.468230 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:30:55.469056 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:55.470157 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:55.471234 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:55.471660 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:55.471784 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:55.477976 (+ 6192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:30:55.477981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:30:55.477981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:30:55.477991 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:30:55.479658 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:55.480155 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:55.480157 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:55.480158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:55.480273 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:55.480593 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:55.480594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:55.480594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:55.480849 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:30:55.480940 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 09:30:55.481254 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 09:30:55.481382 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260926 09:30:55.481501 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:55.467196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40633 (local address 127.0.98.1:35933)
0926 09:30:55.467337 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:55.467351 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:30:55.467417 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:30:55.468262 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:30:55.468265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:30:55.468698 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:30:55.468909 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:55.468916 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:55.470329 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:55.470330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:55.471131 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:55.471135 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:55.471748 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:55.471749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:55.471883 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:55.475935 (+ 4052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:30:55.475940 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:30:55.477421 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:30:55.479661 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:30:55.479669 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:30:55.479672 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:30:55.479947 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:30:55.480361 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:30:55.480362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:30:55.480362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:30:55.480516 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:30:55.480929 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:30:55.480935 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:30:55.481065 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 09:30:55.481177 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 09:30:55.481380 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":56,"thread_start_us":73,"threads_started":1}
I20260926 09:30:55.481518 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:55.467301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35933 (local address 127.0.98.2:40633)
0926 09:30:55.467482 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 09:30:55.467495 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 09:30:55.467522 (+ 27us) server_negotiation.cc:407] Connection header received
0926 09:30:55.467976 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:30:55.467979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:30:55.468023 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:30:55.468159 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:30:55.469061 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:55.470214 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:55.471265 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:30:55.471652 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:30:55.471767 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:55.477563 (+ 5796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:30:55.477567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:30:55.477568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:30:55.477579 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:30:55.479541 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:55.480049 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:55.480052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:55.480053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:55.480184 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:30:55.480601 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:30:55.480602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:30:55.480602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:30:55.480851 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:30:55.480943 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 09:30:55.481247 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 09:30:55.481406 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":64,"threads_started":1}
I20260926 09:30:55.481611 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:55.467196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36181 (local address 127.0.98.1:53167)
0926 09:30:55.467337 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:55.467351 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:30:55.467417 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:30:55.468262 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:30:55.468265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:30:55.468704 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:30:55.468909 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:55.468916 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:55.470259 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:55.470261 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:55.471132 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:30:55.471135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:30:55.471766 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:30:55.471767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:30:55.471878 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:30:55.475926 (+ 4048us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:30:55.475940 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:30:55.477839 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:30:55.479781 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:30:55.479782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:30:55.479783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:30:55.480035 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:30:55.480367 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:30:55.480368 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:30:55.480368 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:30:55.480518 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:30:55.480935 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:30:55.480936 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:30:55.481066 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 09:30:55.481190 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 09:30:55.481460 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":153,"thread_start_us":74,"threads_started":1}
W20260926 09:30:55.482393 16886 leader_election.cc:341] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be0ba63ea67542388ff599a1f1ba7440 (127.0.98.2:40633): Not found: Tablet not found: b9f6262602354c22b032da9ab2e40866
W20260926 09:30:55.482916 16887 leader_election.cc:341] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer daef8174a6bf4c75ab1f7fe0bbd7a9bb (127.0.98.3:36181): Not found: Tablet not found: b9f6262602354c22b032da9ab2e40866
I20260926 09:30:55.482987 16887 leader_election.cc:302] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43448beb01ec4db1bc08f07e25d7f141; no voters: be0ba63ea67542388ff599a1f1ba7440, daef8174a6bf4c75ab1f7fe0bbd7a9bb
I20260926 09:30:55.483228 17272 raft_consensus.cc:2726] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:30:55.738591 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:55.738436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.1:53179)
0926 09:30:55.738499 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:55.738542 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 09:30:55.800619 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:55.800367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.2:47467)
0926 09:30:55.800519 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:55.800570 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260926 09:30:55.947507 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:55.947265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.3:40855)
0926 09:30:55.947414 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:55.947460 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20260926 09:30:56.739359 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:56.739121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.1:58561)
0926 09:30:56.739268 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:56.739312 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
I20260926 09:30:56.801409 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:56.801161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.2:48671)
0926 09:30:56.801323 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:56.801366 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":77,"threads_started":1}
I20260926 09:30:56.948307 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:56.948095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.3:60883)
0926 09:30:56.948222 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:56.948264 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260926 09:30:57.059885 17282 raft_consensus.cc:482] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:30:57.059983 17282 raft_consensus.cc:504] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } }
I20260926 09:30:57.060237 17282 leader_election.cc:288] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers daef8174a6bf4c75ab1f7fe0bbd7a9bb (127.0.98.3:36181), be0ba63ea67542388ff599a1f1ba7440 (127.0.98.2:40633)
W20260926 09:30:57.060627 16886 leader_election.cc:341] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be0ba63ea67542388ff599a1f1ba7440 (127.0.98.2:40633): Not found: Tablet not found: b9f6262602354c22b032da9ab2e40866
W20260926 09:30:57.060796 16887 leader_election.cc:341] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer daef8174a6bf4c75ab1f7fe0bbd7a9bb (127.0.98.3:36181): Not found: Tablet not found: b9f6262602354c22b032da9ab2e40866
I20260926 09:30:57.060879 16887 leader_election.cc:302] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43448beb01ec4db1bc08f07e25d7f141; no voters: be0ba63ea67542388ff599a1f1ba7440, daef8174a6bf4c75ab1f7fe0bbd7a9bb
I20260926 09:30:57.061122 17282 raft_consensus.cc:2726] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:30:57.740178 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:57.739917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.1:41285)
0926 09:30:57.740059 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:57.740133 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
I20260926 09:30:57.802290 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:57.802016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.2:49711)
0926 09:30:57.802187 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:57.802229 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":84,"threads_started":1}
I20260926 09:30:57.949087 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:57.948856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.3:58509)
0926 09:30:57.948996 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:57.949043 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":70,"threads_started":1}
I20260926 09:30:58.741050 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:58.740800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.1:43691)
0926 09:30:58.740960 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:58.741003 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":62,"threads_started":1}
I20260926 09:30:58.803097 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:58.802837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.2:58943)
0926 09:30:58.803004 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:58.803050 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":74,"threads_started":1}
I20260926 09:30:58.950028 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:58.949692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.3:45931)
0926 09:30:58.949929 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:58.949981 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":61,"threads_started":1}
I20260926 09:30:59.741873 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:59.741655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.1:54343)
0926 09:30:59.741780 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:59.741826 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260926 09:30:59.804026 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:59.803784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.2:51313)
0926 09:30:59.803933 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:59.803980 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260926 09:30:59.950721 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:30:59.950522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.3:54075)
0926 09:30:59.950635 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 09:30:59.950680 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260926 09:31:00.144024 17292 raft_consensus.cc:482] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:31:00.144155 17292 raft_consensus.cc:504] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } }
I20260926 09:31:00.144347 17292 leader_election.cc:288] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers daef8174a6bf4c75ab1f7fe0bbd7a9bb (127.0.98.3:36181), be0ba63ea67542388ff599a1f1ba7440 (127.0.98.2:40633)
W20260926 09:31:00.144804 16887 leader_election.cc:341] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer daef8174a6bf4c75ab1f7fe0bbd7a9bb (127.0.98.3:36181): Not found: Tablet not found: b9f6262602354c22b032da9ab2e40866
W20260926 09:31:00.144901 16886 leader_election.cc:341] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be0ba63ea67542388ff599a1f1ba7440 (127.0.98.2:40633): Not found: Tablet not found: b9f6262602354c22b032da9ab2e40866
I20260926 09:31:00.145010 16886 leader_election.cc:302] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43448beb01ec4db1bc08f07e25d7f141; no voters: be0ba63ea67542388ff599a1f1ba7440, daef8174a6bf4c75ab1f7fe0bbd7a9bb
I20260926 09:31:00.145098 17292 raft_consensus.cc:2726] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:31:00.742604 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:00.742396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.1:37707)
0926 09:31:00.742516 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:00.742555 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260926 09:31:00.804752 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:00.804519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.2:37699)
0926 09:31:00.804668 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:00.804703 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
I20260926 09:31:00.951462 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:00.951245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.3:33987)
0926 09:31:00.951384 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:00.951420 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":60,"threads_started":1}
I20260926 09:31:01.743327 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:01.743100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.1:56443)
0926 09:31:01.743235 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:01.743280 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
I20260926 09:31:01.805436 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:01.805203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.2:56489)
0926 09:31:01.805349 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:01.805390 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":65,"threads_started":1}
I20260926 09:31:01.952170 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:01.951942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.3:56389)
0926 09:31:01.952093 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:01.952128 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
I20260926 09:31:02.744113 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:02.743863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.1:35677)
0926 09:31:02.744005 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:02.744061 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260926 09:31:02.806190 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:02.805974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.2:39415)
0926 09:31:02.806109 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:02.806147 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
I20260926 09:31:02.952947 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:02.952717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.3:41705)
0926 09:31:02.952855 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:02.952903 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
I20260926 09:31:03.645684 17302 raft_consensus.cc:482] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:31:03.645792 17302 raft_consensus.cc:504] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daef8174a6bf4c75ab1f7fe0bbd7a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36181 } } peers { permanent_uuid: "be0ba63ea67542388ff599a1f1ba7440" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "43448beb01ec4db1bc08f07e25d7f141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43595 } }
I20260926 09:31:03.645987 17302 leader_election.cc:288] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers daef8174a6bf4c75ab1f7fe0bbd7a9bb (127.0.98.3:36181), be0ba63ea67542388ff599a1f1ba7440 (127.0.98.2:40633)
W20260926 09:31:03.646373 16887 leader_election.cc:341] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer daef8174a6bf4c75ab1f7fe0bbd7a9bb (127.0.98.3:36181): Not found: Tablet not found: b9f6262602354c22b032da9ab2e40866
W20260926 09:31:03.646423 16886 leader_election.cc:341] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be0ba63ea67542388ff599a1f1ba7440 (127.0.98.2:40633): Not found: Tablet not found: b9f6262602354c22b032da9ab2e40866
I20260926 09:31:03.646502 16886 leader_election.cc:302] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43448beb01ec4db1bc08f07e25d7f141; no voters: be0ba63ea67542388ff599a1f1ba7440, daef8174a6bf4c75ab1f7fe0bbd7a9bb
I20260926 09:31:03.646605 17302 raft_consensus.cc:2726] T b9f6262602354c22b032da9ab2e40866 P 43448beb01ec4db1bc08f07e25d7f141 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:31:03.744889 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:03.744661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.1:60275)
0926 09:31:03.744798 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:03.744843 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260926 09:31:03.806949 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:03.806724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.2:41865)
0926 09:31:03.806867 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:03.806905 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260926 09:31:03.953750 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:03.953433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.3:45337)
0926 09:31:03.953667 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:03.953706 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":138,"threads_started":1}
I20260926 09:31:04.745754 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:04.745441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.1:36751)
0926 09:31:04.745653 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:04.745707 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":63,"threads_started":1}
I20260926 09:31:04.807636 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:04.807415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.2:33303)
0926 09:31:04.807551 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:04.807592 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
I20260926 09:31:04.954499 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:04.954268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.3:44383)
0926 09:31:04.954396 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:04.954442 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20260926 09:31:05.746430 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:05.746189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.1:37405)
0926 09:31:05.746344 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:05.746383 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":67,"threads_started":1}
I20260926 09:31:05.808415 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:05.808173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.2:57713)
0926 09:31:05.808300 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:05.808346 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260926 09:31:05.955261 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:05.955042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.3:55669)
0926 09:31:05.955173 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:05.955217 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260926 09:31:06.747372 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:06.747125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.1:48513)
0926 09:31:06.747290 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:06.747325 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
I20260926 09:31:06.809290 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:06.809038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.2:56813)
0926 09:31:06.809186 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:06.809242 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":64,"threads_started":1}
I20260926 09:31:06.956032 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:06.955731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.3:44053)
0926 09:31:06.955935 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:06.955985 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":61,"threads_started":1}
I20260926 09:31:07.748208 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:07.747966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.1:43439)
0926 09:31:07.748115 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:07.748161 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":66,"threads_started":1}
I20260926 09:31:07.810081 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:07.809845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.2:47633)
0926 09:31:07.809983 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:07.810031 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
I20260926 09:31:07.956727 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:07.956514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.3:56019)
0926 09:31:07.956647 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:07.956681 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20260926 09:31:08.749181 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:08.748922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.1:55011)
0926 09:31:08.749085 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:08.749135 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":61,"threads_started":1}
I20260926 09:31:08.810988 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:08.810722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.2:54869)
0926 09:31:08.810894 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:08.810941 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":79,"threads_started":1}
I20260926 09:31:08.957640 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:08.957342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39239 (local address 127.0.98.3:35195)
0926 09:31:08.957514 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:08.957565 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":82,"threads_started":1}
I20260926 09:31:08.964353 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 16835
I20260926 09:31:08.968916 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 17004
I20260926 09:31:08.975618 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 17136
2026-09-26T09:31:08Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58326 ms)
[----------] 30 tests from SecurityITest (229915 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:31:09 dist-test-slave-dfzm krb5kdc[17323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:31:09 dist-test-slave-dfzm krb5kdc[17323](info): set up 2 sockets
Sep 26 09:31:09 dist-test-slave-dfzm krb5kdc[17323](info): commencing operation
krb5kdc: starting...
W20260926 09:31:10.093250 392 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:31:10 dist-test-slave-dfzm krb5kdc[17323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:31:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:31:10.214057 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37785
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:10.296593 17336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:10.296813 17336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:10.296835 17336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:10.297974 17336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:10.298014 17336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:10.298028 17336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:10.298041 17336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:10.298053 17336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:10.299624 17336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:10.299825 17336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:10.300004 17336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:10.305357 17341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:10.305416 17342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:10.305434 17336 server_base.cc:1034] running on GCE node
W20260926 09:31:10.305519 17344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:10.305898 17336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:10.306211 17336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:10.307397 17336 hybrid_clock.cc:648] HybridClock initialized: now 1790415070307383 us; error 53 us; skew 500 ppm
Sep 26 09:31:10 dist-test-slave-dfzm krb5kdc[17323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:10.310070 17336 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:31:10.310534 17336 webserver.cc:460] Webserver started at http://127.0.98.62:40327/ using document root <none> and password file <none>
I20260926 09:31:10.310760 17336 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:10.310827 17336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:10.310979 17336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:10.312041 17336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "8aeac50451374948a366fdee1687fc8b"
format_stamp: "Formatted at 2026-09-26 09:31:10 on dist-test-slave-dfzm"
server_key: "89d3e52f6e11f12770031fd6087186d4"
server_key_iv: "73cfd8e275163cf39c6cc4d3c73f4275"
server_key_version: "encryptionkey@0"
I20260926 09:31:10.312459 17336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "8aeac50451374948a366fdee1687fc8b"
format_stamp: "Formatted at 2026-09-26 09:31:10 on dist-test-slave-dfzm"
server_key: "89d3e52f6e11f12770031fd6087186d4"
server_key_iv: "73cfd8e275163cf39c6cc4d3c73f4275"
server_key_version: "encryptionkey@0"
I20260926 09:31:10.313848 17336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:31:10.314535 17351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:10.314699 17336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:31:10.314798 17336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "8aeac50451374948a366fdee1687fc8b"
format_stamp: "Formatted at 2026-09-26 09:31:10 on dist-test-slave-dfzm"
server_key: "89d3e52f6e11f12770031fd6087186d4"
server_key_iv: "73cfd8e275163cf39c6cc4d3c73f4275"
server_key_version: "encryptionkey@0"
I20260926 09:31:10.314872 17336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:10.324105 17336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:10.326089 17336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:10.326251 17336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:10.330094 17336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37785
I20260926 09:31:10.330129 17403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37785 every 8 connection(s)
I20260926 09:31:10.330502 17336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:10.331393 17404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:10.333701 17404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aeac50451374948a366fdee1687fc8b: Bootstrap starting.
I20260926 09:31:10.334244 17404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8aeac50451374948a366fdee1687fc8b: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:10.334482 17404 log.cc:824] T 00000000000000000000000000000000 P 8aeac50451374948a366fdee1687fc8b: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:10.335219 17404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aeac50451374948a366fdee1687fc8b: No bootstrap required, opened a new log
I20260926 09:31:10.337374 17404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8aeac50451374948a366fdee1687fc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aeac50451374948a366fdee1687fc8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37785 } }
I20260926 09:31:10.337517 17404 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8aeac50451374948a366fdee1687fc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:10.337539 17404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8aeac50451374948a366fdee1687fc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aeac50451374948a366fdee1687fc8b, State: Initialized, Role: FOLLOWER
I20260926 09:31:10.337743 17404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8aeac50451374948a366fdee1687fc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aeac50451374948a366fdee1687fc8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37785 } }
I20260926 09:31:10.337842 17404 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8aeac50451374948a366fdee1687fc8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:31:10.337888 17404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8aeac50451374948a366fdee1687fc8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:31:10.337936 17404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8aeac50451374948a366fdee1687fc8b [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:10.338483 17404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8aeac50451374948a366fdee1687fc8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aeac50451374948a366fdee1687fc8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37785 } }
I20260926 09:31:10.338572 17404 leader_election.cc:302] T 00000000000000000000000000000000 P 8aeac50451374948a366fdee1687fc8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8aeac50451374948a366fdee1687fc8b; no voters:
I20260926 09:31:10.338766 17404 leader_election.cc:288] T 00000000000000000000000000000000 P 8aeac50451374948a366fdee1687fc8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:31:10.339028 17404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8aeac50451374948a366fdee1687fc8b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:10.339041 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 17336
I20260926 09:31:10.339159 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:10.339357 392 external_mini_cluster.cc:1428] Setting key a3f9cf05443bdb0d5a2935fc225bacfe
I20260926 09:31:10.339572 17407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8aeac50451374948a366fdee1687fc8b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:10.339747 17407 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8aeac50451374948a366fdee1687fc8b [term 1 LEADER]: Becoming Leader. State: Replica: 8aeac50451374948a366fdee1687fc8b, State: Running, Role: LEADER
I20260926 09:31:10.339859 17407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8aeac50451374948a366fdee1687fc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aeac50451374948a366fdee1687fc8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37785 } }
I20260926 09:31:10.340251 17407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aeac50451374948a366fdee1687fc8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8aeac50451374948a366fdee1687fc8b. Latest consensus state: current_term: 1 leader_uuid: "8aeac50451374948a366fdee1687fc8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aeac50451374948a366fdee1687fc8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37785 } } }
I20260926 09:31:10.340332 17407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aeac50451374948a366fdee1687fc8b [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:10.340370 17408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aeac50451374948a366fdee1687fc8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8aeac50451374948a366fdee1687fc8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aeac50451374948a366fdee1687fc8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37785 } } }
I20260926 09:31:10.340481 17408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aeac50451374948a366fdee1687fc8b [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:10.340610 17424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:10.341221 17424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:10.342975 17424 catalog_manager.cc:1347] Generated new cluster ID: 596084c1cb174b2299f77a611c6e8212
I20260926 09:31:10.343034 17424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:31:10 dist-test-slave-dfzm krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:31:10.349802 17424 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:10.350600 17424 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:10.355835 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:10.340229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36810 (local address 127.0.98.62:37785)
0926 09:31:10.341160 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:10.341169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:10.341208 (+ 39us) server_negotiation.cc:407] Connection header received
0926 09:31:10.341330 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:10.341338 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:10.341395 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:10.341545 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:10.342312 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.343462 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.344270 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.344708 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.344842 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:10.349756 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:10.349762 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:10.349762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:10.349779 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:10.352853 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:10.353629 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:10.353633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:10.353634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:10.353783 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:10.354070 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:10.354070 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:10.354071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:10.354413 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:10.354824 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:10.355345 (+ 521us) server_negotiation.cc:299] Negotiation successful
0926 09:31:10.355526 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":62,"threads_started":1}
I20260926 09:31:10.358026 17424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8aeac50451374948a366fdee1687fc8b: Generated new TSK 0
I20260926 09:31:10.358189 17424 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:31:10.395946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37785
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:10.485639 17430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:10.485899 17430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:10.485944 17430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:10.487156 17430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:10.487224 17430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:10.487272 17430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:31:10.488816 17430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:10.489089 17430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:10.489324 17430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:10.495025 17436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:10.495049 17438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:10.495041 17435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:10.495448 17430 server_base.cc:1034] running on GCE node
I20260926 09:31:10.495656 17430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:10.495903 17430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:10.497069 17430 hybrid_clock.cc:648] HybridClock initialized: now 1790415070497053 us; error 27 us; skew 500 ppm
Sep 26 09:31:10 dist-test-slave-dfzm krb5kdc[17323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:10.499971 17430 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:31:10.500455 17430 webserver.cc:460] Webserver started at http://127.0.98.1:37291/ using document root <none> and password file <none>
I20260926 09:31:10.500679 17430 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:10.500746 17430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:10.500906 17430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:10.502043 17430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "32a627ccc8f4473da4bf47168d982c15"
format_stamp: "Formatted at 2026-09-26 09:31:10 on dist-test-slave-dfzm"
server_key: "15132b065255f712a84324378d02bf13"
server_key_iv: "636dfb7998e88a2a2aa77ab2cc77a804"
server_key_version: "encryptionkey@0"
I20260926 09:31:10.502409 17430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "32a627ccc8f4473da4bf47168d982c15"
format_stamp: "Formatted at 2026-09-26 09:31:10 on dist-test-slave-dfzm"
server_key: "15132b065255f712a84324378d02bf13"
server_key_iv: "636dfb7998e88a2a2aa77ab2cc77a804"
server_key_version: "encryptionkey@0"
I20260926 09:31:10.503721 17430 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:31:10.504441 17445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:10.504611 17430 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:31:10.504714 17430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "32a627ccc8f4473da4bf47168d982c15"
format_stamp: "Formatted at 2026-09-26 09:31:10 on dist-test-slave-dfzm"
server_key: "15132b065255f712a84324378d02bf13"
server_key_iv: "636dfb7998e88a2a2aa77ab2cc77a804"
server_key_version: "encryptionkey@0"
I20260926 09:31:10.504792 17430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:10.520810 17430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:10.522938 17430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:10.523109 17430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:10.523357 17430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:31:10.523700 17430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:31:10.523757 17430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:10.523794 17430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:31:10.523825 17430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:10.529537 17430 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46205
I20260926 09:31:10.529642 17558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46205 every 8 connection(s)
I20260926 09:31:10.529964 17430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:31:10.532014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 17430
I20260926 09:31:10.532160 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:31:10.532404 392 external_mini_cluster.cc:1428] Setting key 3f39012c787fdd3882690e1da7289539
Sep 26 09:31:10 dist-test-slave-dfzm krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:31:10.540987 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:10.530727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41759 (local address 127.0.98.62:37785)
0926 09:31:10.530780 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:10.530782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:10.531014 (+ 232us) server_negotiation.cc:407] Connection header received
0926 09:31:10.532182 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:10.532183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:10.532198 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:10.532256 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:10.533101 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.533672 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.534696 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.534922 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.535009 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:10.537663 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:10.537667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:10.537667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:10.537679 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:10.539307 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:10.539848 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:10.539850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:10.539850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:10.539910 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:10.540284 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:10.540285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:10.540285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:10.540412 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:10.540472 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:10.540844 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 09:31:10.540921 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 09:31:10.541282 17561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:10.530688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37785 (local address 127.0.98.1:41759)
0926 09:31:10.530871 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:10.530917 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:10.532024 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:10.532388 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:10.532391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:10.532790 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:10.532986 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:10.532991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.533765 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.533766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:10.534596 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:10.534598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.535020 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.535021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:10.535150 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:10.535719 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:10.535727 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:10.537509 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:10.539397 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:10.539406 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:10.539409 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:10.539749 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:10.539991 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:10.539992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:10.539992 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:10.540206 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:10.540488 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:10.540495 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:10.540681 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 09:31:10.540830 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 09:31:10.541018 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":110,"threads_started":1}
I20260926 09:31:10.541949 17559 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37785
I20260926 09:31:10.542037 17559 heartbeater.cc:464] Registering TS with master...
I20260926 09:31:10.542186 17559 heartbeater.cc:511] Master 127.0.98.62:37785 requested a full tablet report, sending...
I20260926 09:31:10.542689 17368 ts_manager.cc:194] Registered new tserver with Master: 32a627ccc8f4473da4bf47168d982c15 (127.0.98.1:46205)
I20260926 09:31:10.543326 17368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41759
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:31:10.560540 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37785
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:10.641371 17564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:10.641641 17564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:10.641726 17564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:10.642905 17564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:10.642970 17564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:10.643016 17564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:31:10.644546 17564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:10.644824 17564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:10.645069 17564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:10.649986 17569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:10.649997 17572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:10.649997 17570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:10.650255 17564 server_base.cc:1034] running on GCE node
I20260926 09:31:10.650482 17564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:10.650746 17564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:10.651914 17564 hybrid_clock.cc:648] HybridClock initialized: now 1790415070651916 us; error 42 us; skew 500 ppm
Sep 26 09:31:10 dist-test-slave-dfzm krb5kdc[17323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:10.654438 17564 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:31:10.654872 17564 webserver.cc:460] Webserver started at http://127.0.98.2:42903/ using document root <none> and password file <none>
I20260926 09:31:10.655112 17564 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:10.655189 17564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:10.655339 17564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:10.656471 17564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "744a758dc8934ef2a91799ae930d3b07"
format_stamp: "Formatted at 2026-09-26 09:31:10 on dist-test-slave-dfzm"
server_key: "d350e2d406d3281337e828951b0945ac"
server_key_iv: "fe850927968a06b99cd014fe5d6ee3c6"
server_key_version: "encryptionkey@0"
I20260926 09:31:10.656840 17564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "744a758dc8934ef2a91799ae930d3b07"
format_stamp: "Formatted at 2026-09-26 09:31:10 on dist-test-slave-dfzm"
server_key: "d350e2d406d3281337e828951b0945ac"
server_key_iv: "fe850927968a06b99cd014fe5d6ee3c6"
server_key_version: "encryptionkey@0"
I20260926 09:31:10.658322 17564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 09:31:10.659117 17579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:10.659325 17564 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:10.659431 17564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "744a758dc8934ef2a91799ae930d3b07"
format_stamp: "Formatted at 2026-09-26 09:31:10 on dist-test-slave-dfzm"
server_key: "d350e2d406d3281337e828951b0945ac"
server_key_iv: "fe850927968a06b99cd014fe5d6ee3c6"
server_key_version: "encryptionkey@0"
I20260926 09:31:10.659505 17564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:10.669744 17564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:10.671669 17564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:10.671846 17564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:10.672104 17564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:31:10.672415 17564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:31:10.672469 17564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:10.672504 17564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:31:10.672552 17564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:10.678395 17564 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37865
I20260926 09:31:10.678437 17692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37865 every 8 connection(s)
I20260926 09:31:10.678756 17564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:31:10 dist-test-slave-dfzm krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:31:10.685376 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 17564
I20260926 09:31:10.685508 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:31:10.685786 392 external_mini_cluster.cc:1428] Setting key f97ac8fe2cf902391dc202bf31236f86
I20260926 09:31:10.688190 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:10.679457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53505 (local address 127.0.98.62:37785)
0926 09:31:10.679523 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:10.679525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:10.679735 (+ 210us) server_negotiation.cc:407] Connection header received
0926 09:31:10.680603 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:10.680604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:10.680615 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:10.680679 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:10.681250 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.681821 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.682778 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.682962 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.683020 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:10.685043 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:10.685045 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:10.685045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:10.685052 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:10.686615 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:10.687079 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:10.687080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:10.687081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:10.687148 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:10.687405 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:10.687406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:10.687407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:10.687566 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:10.687655 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:10.687977 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 09:31:10.688072 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 09:31:10.688232 17695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:10.679443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37785 (local address 127.0.98.2:53505)
0926 09:31:10.679599 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:10.679633 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:10.680512 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:10.680734 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:10.680738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:10.680992 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:10.681175 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:10.681179 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.681899 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.681900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:10.682703 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:10.682704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.683038 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.683038 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:10.683132 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:10.683643 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:10.683651 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:10.684963 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:10.686747 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:10.686754 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:10.686756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:10.686987 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:10.687212 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:10.687213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:10.687214 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:10.687336 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:10.687648 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:10.687655 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:10.687831 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 09:31:10.687936 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 09:31:10.688070 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260926 09:31:10.688733 17693 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37785
I20260926 09:31:10.688819 17693 heartbeater.cc:464] Registering TS with master...
I20260926 09:31:10.689038 17693 heartbeater.cc:511] Master 127.0.98.62:37785 requested a full tablet report, sending...
I20260926 09:31:10.689455 17368 ts_manager.cc:194] Registered new tserver with Master: 744a758dc8934ef2a91799ae930d3b07 (127.0.98.2:37865)
I20260926 09:31:10.689870 17368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53505
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:31:10.713212 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37785
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:10.793664 17698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:10.793871 17698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:10.793895 17698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:10.795071 17698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:10.795117 17698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:10.795145 17698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:31:10.796610 17698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:10.796837 17698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:10.797079 17698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:10.801973 17703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:10.801975 17706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:10.801975 17704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:10.802186 17698 server_base.cc:1034] running on GCE node
I20260926 09:31:10.802484 17698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:10.802788 17698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:10.803970 17698 hybrid_clock.cc:648] HybridClock initialized: now 1790415070803972 us; error 43 us; skew 500 ppm
Sep 26 09:31:10 dist-test-slave-dfzm krb5kdc[17323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:10.806635 17698 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:31:10.807119 17698 webserver.cc:460] Webserver started at http://127.0.98.3:36553/ using document root <none> and password file <none>
I20260926 09:31:10.807399 17698 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:10.807469 17698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:10.807616 17698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:10.808717 17698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "57770afb759e454782060a6a1391037c"
format_stamp: "Formatted at 2026-09-26 09:31:10 on dist-test-slave-dfzm"
server_key: "8921d15971288bbfe1c5e43331301fc2"
server_key_iv: "047185e3095d0899e6eb80376c8c8d1a"
server_key_version: "encryptionkey@0"
I20260926 09:31:10.809082 17698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "57770afb759e454782060a6a1391037c"
format_stamp: "Formatted at 2026-09-26 09:31:10 on dist-test-slave-dfzm"
server_key: "8921d15971288bbfe1c5e43331301fc2"
server_key_iv: "047185e3095d0899e6eb80376c8c8d1a"
server_key_version: "encryptionkey@0"
I20260926 09:31:10.810487 17698 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:31:10.811337 17713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:10.811545 17698 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:31:10.811674 17698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "57770afb759e454782060a6a1391037c"
format_stamp: "Formatted at 2026-09-26 09:31:10 on dist-test-slave-dfzm"
server_key: "8921d15971288bbfe1c5e43331301fc2"
server_key_iv: "047185e3095d0899e6eb80376c8c8d1a"
server_key_version: "encryptionkey@0"
I20260926 09:31:10.811769 17698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:10.821154 17698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:10.823195 17698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:10.823369 17698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:10.823612 17698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:31:10.823920 17698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:31:10.823977 17698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:10.824054 17698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:31:10.824088 17698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:10.830714 17698 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46467
I20260926 09:31:10.830754 17826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46467 every 8 connection(s)
I20260926 09:31:10.831110 17698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:31:10 dist-test-slave-dfzm krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:31:10.838006 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 17698
I20260926 09:31:10.838140 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:31:10.838321 392 external_mini_cluster.cc:1428] Setting key a30bfb735b02a195cbefce191b1a35e8
I20260926 09:31:10.841676 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:10.831903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33303 (local address 127.0.98.62:37785)
0926 09:31:10.831987 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:10.831990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:10.832183 (+ 193us) server_negotiation.cc:407] Connection header received
0926 09:31:10.833192 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:10.833193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:10.833204 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:10.833261 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:10.833961 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.834458 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.835592 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.835794 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.835864 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:10.838118 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:10.838120 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:10.838120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:10.838128 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:10.840020 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:10.840405 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:10.840406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:10.840406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:10.840466 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:10.840892 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:10.840893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:10.840893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:10.841012 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:10.841078 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:10.841404 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 09:31:10.841543 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 09:31:10.841675 17829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:10.831849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37785 (local address 127.0.98.3:33303)
0926 09:31:10.832057 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:10.832094 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:10.833089 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:10.833322 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:10.833325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:10.833663 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:10.833874 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:10.833879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.834560 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.834560 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:10.835489 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:10.835492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.835909 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.835910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:10.836027 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:10.836522 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:10.836530 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:10.837983 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:10.840092 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:10.840098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:10.840100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:10.840327 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:10.840662 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:10.840663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:10.840664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:10.840804 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:10.841079 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:10.841084 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:10.841221 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 09:31:10.841310 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 09:31:10.841434 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":109,"threads_started":1}
I20260926 09:31:10.842123 17827 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37785
I20260926 09:31:10.842226 17827 heartbeater.cc:464] Registering TS with master...
I20260926 09:31:10.842404 17827 heartbeater.cc:511] Master 127.0.98.62:37785 requested a full tablet report, sending...
I20260926 09:31:10.842779 17368 ts_manager.cc:194] Registered new tserver with Master: 57770afb759e454782060a6a1391037c (127.0.98.3:46467)
I20260926 09:31:10.843170 17368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33303
I20260926 09:31:10.850421 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:31:10.858707 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:10.852006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36824 (local address 127.0.98.62:37785)
0926 09:31:10.852063 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:10.852065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:10.852139 (+ 74us) server_negotiation.cc:407] Connection header received
0926 09:31:10.852190 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:10.852191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:10.852200 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:10.852257 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:10.852846 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.853364 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.854250 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.854435 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.854507 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:10.855875 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:10.855876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:10.855877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:10.855883 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:10.857346 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:10.857705 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:10.857706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:10.857706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:10.857756 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:10.858009 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:10.858009 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:10.858010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:10.858237 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:10.858313 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:10.858568 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 09:31:10.858645 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 09:31:10.860342 17368 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:31:10.861989 17368 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:31:10.868630 17839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:10.864253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46205 (local address 127.0.0.1:34482)
0926 09:31:10.864435 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:10.864478 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:10.864544 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:10.865023 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:10.865026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:10.865029 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:10.865138 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:10.865139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.866192 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.866193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:10.867348 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:10.867351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.868343 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.868344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:10.868423 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:10.868427 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:31:10.868461 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 09:31:10.868527 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":10,"thread_start_us":68,"threads_started":1}
I20260926 09:31:10.868631 17840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:10.864418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37804 (local address 127.0.98.2:37865)
0926 09:31:10.864808 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:10.864813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:10.864826 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:31:10.864870 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:10.864873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:10.864892 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:10.865248 (+ 356us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:10.865340 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.866561 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.867735 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.868224 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.868314 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:10.868372 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:10.868517 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 09:31:10.868554 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":93,"threads_started":1}
I20260926 09:31:10.868773 17841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:10.864288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37865 (local address 127.0.0.1:37804)
0926 09:31:10.864611 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:10.864623 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:10.864681 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:10.865023 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:10.865026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:10.865031 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:10.865137 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:10.865139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.866676 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.866677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:10.867644 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:10.867645 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.868314 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.868314 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:10.868393 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:10.868406 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 09:31:10.868500 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 09:31:10.868544 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":22,"thread_start_us":96,"threads_started":1}
I20260926 09:31:10.869483 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:10.864466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46467 (local address 127.0.0.1:42028)
0926 09:31:10.865124 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:10.865135 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:10.865188 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:10.865413 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:10.865413 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:10.865415 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:10.865465 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:10.865466 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.866828 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.866829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:10.868137 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:10.868141 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.869239 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.869240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:10.869304 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:10.869308 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:31:10.869365 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 09:31:10.869405 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":61,"threads_started":1}
I20260926 09:31:10.869518 17627 tablet_service.cc:1467] Processing CreateTablet for tablet a28c94ba2a244e79a471d705a8fb722a (DEFAULT_TABLE table=test-table [id=0f25c6dd258845ffb43b716eb8dc4dd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:31:10.869747 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:10.864661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34482 (local address 127.0.98.1:46205)
0926 09:31:10.864861 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:10.864865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:10.864874 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:31:10.864913 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:10.864917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:10.864933 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:10.865017 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:10.865233 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.866088 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.867437 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.868249 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.868608 (+ 359us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:10.868676 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:10.869247 (+ 571us) server_negotiation.cc:299] Negotiation successful
0926 09:31:10.869294 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":85,"threads_started":1}
I20260926 09:31:10.869781 17842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:10.864604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42028 (local address 127.0.98.3:46467)
0926 09:31:10.864789 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:10.864794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:10.865258 (+ 464us) server_negotiation.cc:407] Connection header received
0926 09:31:10.865311 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:10.865314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:10.865332 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:10.865416 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:10.865543 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.866729 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.868249 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:10.869058 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:10.869134 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:10.869200 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:10.869633 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 09:31:10.869689 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":119,"threads_started":1}
I20260926 09:31:10.869928 17627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a28c94ba2a244e79a471d705a8fb722a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:10.870353 17493 tablet_service.cc:1467] Processing CreateTablet for tablet a28c94ba2a244e79a471d705a8fb722a (DEFAULT_TABLE table=test-table [id=0f25c6dd258845ffb43b716eb8dc4dd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:31:10.870675 17493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a28c94ba2a244e79a471d705a8fb722a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:10.872565 17844 tablet_bootstrap.cc:492] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07: Bootstrap starting.
I20260926 09:31:10.873144 17761 tablet_service.cc:1467] Processing CreateTablet for tablet a28c94ba2a244e79a471d705a8fb722a (DEFAULT_TABLE table=test-table [id=0f25c6dd258845ffb43b716eb8dc4dd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:31:10.873302 17844 tablet_bootstrap.cc:654] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:10.873440 17761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a28c94ba2a244e79a471d705a8fb722a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:10.873607 17845 tablet_bootstrap.cc:492] T a28c94ba2a244e79a471d705a8fb722a P 32a627ccc8f4473da4bf47168d982c15: Bootstrap starting.
I20260926 09:31:10.873972 17844 log.cc:824] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:10.874282 17845 tablet_bootstrap.cc:654] T a28c94ba2a244e79a471d705a8fb722a P 32a627ccc8f4473da4bf47168d982c15: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:10.874768 17844 tablet_bootstrap.cc:492] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07: No bootstrap required, opened a new log
I20260926 09:31:10.874825 17845 log.cc:824] T a28c94ba2a244e79a471d705a8fb722a P 32a627ccc8f4473da4bf47168d982c15: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:10.874861 17844 ts_tablet_manager.cc:1397] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 09:31:10.875604 17845 tablet_bootstrap.cc:492] T a28c94ba2a244e79a471d705a8fb722a P 32a627ccc8f4473da4bf47168d982c15: No bootstrap required, opened a new log
I20260926 09:31:10.875679 17845 ts_tablet_manager.cc:1397] T a28c94ba2a244e79a471d705a8fb722a P 32a627ccc8f4473da4bf47168d982c15: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:31:10.876734 17848 tablet_bootstrap.cc:492] T a28c94ba2a244e79a471d705a8fb722a P 57770afb759e454782060a6a1391037c: Bootstrap starting.
I20260926 09:31:10.876628 17844 raft_consensus.cc:348] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a627ccc8f4473da4bf47168d982c15" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46205 } } peers { permanent_uuid: "57770afb759e454782060a6a1391037c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46467 } } peers { permanent_uuid: "744a758dc8934ef2a91799ae930d3b07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } }
I20260926 09:31:10.876824 17844 raft_consensus.cc:374] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:10.876868 17844 raft_consensus.cc:729] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744a758dc8934ef2a91799ae930d3b07, State: Initialized, Role: FOLLOWER
I20260926 09:31:10.877015 17844 consensus_queue.cc:260] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a627ccc8f4473da4bf47168d982c15" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46205 } } peers { permanent_uuid: "57770afb759e454782060a6a1391037c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46467 } } peers { permanent_uuid: "744a758dc8934ef2a91799ae930d3b07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } }
I20260926 09:31:10.877354 17844 ts_tablet_manager.cc:1428] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 09:31:10.877501 17693 heartbeater.cc:503] Master 127.0.98.62:37785 was elected leader, sending a full tablet report...
I20260926 09:31:10.877646 17848 tablet_bootstrap.cc:654] T a28c94ba2a244e79a471d705a8fb722a P 57770afb759e454782060a6a1391037c: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:10.878183 17845 raft_consensus.cc:348] T a28c94ba2a244e79a471d705a8fb722a P 32a627ccc8f4473da4bf47168d982c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a627ccc8f4473da4bf47168d982c15" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46205 } } peers { permanent_uuid: "57770afb759e454782060a6a1391037c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46467 } } peers { permanent_uuid: "744a758dc8934ef2a91799ae930d3b07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } }
I20260926 09:31:10.878288 17848 log.cc:824] T a28c94ba2a244e79a471d705a8fb722a P 57770afb759e454782060a6a1391037c: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:10.878332 17845 raft_consensus.cc:374] T a28c94ba2a244e79a471d705a8fb722a P 32a627ccc8f4473da4bf47168d982c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:10.878376 17845 raft_consensus.cc:729] T a28c94ba2a244e79a471d705a8fb722a P 32a627ccc8f4473da4bf47168d982c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32a627ccc8f4473da4bf47168d982c15, State: Initialized, Role: FOLLOWER
I20260926 09:31:10.878675 17845 consensus_queue.cc:260] T a28c94ba2a244e79a471d705a8fb722a P 32a627ccc8f4473da4bf47168d982c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a627ccc8f4473da4bf47168d982c15" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46205 } } peers { permanent_uuid: "57770afb759e454782060a6a1391037c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46467 } } peers { permanent_uuid: "744a758dc8934ef2a91799ae930d3b07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } }
I20260926 09:31:10.878986 17559 heartbeater.cc:503] Master 127.0.98.62:37785 was elected leader, sending a full tablet report...
I20260926 09:31:10.879047 17848 tablet_bootstrap.cc:492] T a28c94ba2a244e79a471d705a8fb722a P 57770afb759e454782060a6a1391037c: No bootstrap required, opened a new log
I20260926 09:31:10.879119 17848 ts_tablet_manager.cc:1397] T a28c94ba2a244e79a471d705a8fb722a P 57770afb759e454782060a6a1391037c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 09:31:10.879297 17845 ts_tablet_manager.cc:1428] T a28c94ba2a244e79a471d705a8fb722a P 32a627ccc8f4473da4bf47168d982c15: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 09:31:10.880740 17848 raft_consensus.cc:348] T a28c94ba2a244e79a471d705a8fb722a P 57770afb759e454782060a6a1391037c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a627ccc8f4473da4bf47168d982c15" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46205 } } peers { permanent_uuid: "57770afb759e454782060a6a1391037c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46467 } } peers { permanent_uuid: "744a758dc8934ef2a91799ae930d3b07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } }
I20260926 09:31:10.880901 17848 raft_consensus.cc:374] T a28c94ba2a244e79a471d705a8fb722a P 57770afb759e454782060a6a1391037c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:10.880955 17848 raft_consensus.cc:729] T a28c94ba2a244e79a471d705a8fb722a P 57770afb759e454782060a6a1391037c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57770afb759e454782060a6a1391037c, State: Initialized, Role: FOLLOWER
I20260926 09:31:10.881071 17848 consensus_queue.cc:260] T a28c94ba2a244e79a471d705a8fb722a P 57770afb759e454782060a6a1391037c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a627ccc8f4473da4bf47168d982c15" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46205 } } peers { permanent_uuid: "57770afb759e454782060a6a1391037c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46467 } } peers { permanent_uuid: "744a758dc8934ef2a91799ae930d3b07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } }
I20260926 09:31:10.881389 17848 ts_tablet_manager.cc:1428] T a28c94ba2a244e79a471d705a8fb722a P 57770afb759e454782060a6a1391037c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 09:31:10.881410 17827 heartbeater.cc:503] Master 127.0.98.62:37785 was elected leader, sending a full tablet report...
W20260926 09:31:10.929304 17694 tablet.cc:2369] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:31:11.011658 17849 raft_consensus.cc:482] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:31:11.011823 17849 raft_consensus.cc:504] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a627ccc8f4473da4bf47168d982c15" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46205 } } peers { permanent_uuid: "57770afb759e454782060a6a1391037c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46467 } } peers { permanent_uuid: "744a758dc8934ef2a91799ae930d3b07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } }
I20260926 09:31:11.012226 17849 leader_election.cc:288] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32a627ccc8f4473da4bf47168d982c15 (127.0.98.1:46205), 57770afb759e454782060a6a1391037c (127.0.98.3:46467)
I20260926 09:31:11.016391 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.012530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50491 (local address 127.0.98.1:46205)
0926 09:31:11.012607 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:11.012610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:11.012621 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:31:11.012759 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:11.012760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:11.012773 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:11.012865 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:11.013054 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.013918 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.015404 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.016004 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.016084 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:11.016094 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:11.016281 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 09:31:11.016319 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 09:31:11.016391 17842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.012699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41283 (local address 127.0.98.3:46467)
0926 09:31:11.012751 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:11.012753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:11.012842 (+ 89us) server_negotiation.cc:407] Connection header received
0926 09:31:11.012895 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:11.012896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:11.012905 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:11.012957 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:11.013113 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.013895 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.015404 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.016013 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.016084 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:11.016094 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:11.016281 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 09:31:11.016320 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:31:11.016453 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.012614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46467 (local address 127.0.98.2:41283)
0926 09:31:11.012774 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:11.012783 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:11.012835 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:11.012972 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:11.012973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:11.012975 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:11.013040 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:11.013041 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.013990 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.013991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:11.015304 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:11.015308 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.016102 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.016103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:11.016190 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:11.016194 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:31:11.016265 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 09:31:11.016319 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":50,"thread_start_us":104,"threads_started":1}
I20260926 09:31:11.016446 17695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.012473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46205 (local address 127.0.98.2:50491)
0926 09:31:11.012549 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:11.012564 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:11.012628 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:11.012873 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:11.012874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:11.012876 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:11.012967 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:11.012969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.013990 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.013991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:11.015304 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:11.015308 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.016102 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.016103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:11.016197 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:11.016199 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:31:11.016265 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 09:31:11.016320 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":57}
I20260926 09:31:11.016865 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28c94ba2a244e79a471d705a8fb722a" candidate_uuid: "744a758dc8934ef2a91799ae930d3b07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57770afb759e454782060a6a1391037c" is_pre_election: true
I20260926 09:31:11.017027 17781 raft_consensus.cc:2454] T a28c94ba2a244e79a471d705a8fb722a P 57770afb759e454782060a6a1391037c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744a758dc8934ef2a91799ae930d3b07 in term 0.
I20260926 09:31:11.017011 17513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28c94ba2a244e79a471d705a8fb722a" candidate_uuid: "744a758dc8934ef2a91799ae930d3b07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32a627ccc8f4473da4bf47168d982c15" is_pre_election: true
I20260926 09:31:11.017143 17513 raft_consensus.cc:2454] T a28c94ba2a244e79a471d705a8fb722a P 32a627ccc8f4473da4bf47168d982c15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 744a758dc8934ef2a91799ae930d3b07 in term 0.
I20260926 09:31:11.017271 17580 leader_election.cc:302] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57770afb759e454782060a6a1391037c, 744a758dc8934ef2a91799ae930d3b07; no voters:
I20260926 09:31:11.017431 17849 raft_consensus.cc:2781] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:31:11.017489 17849 raft_consensus.cc:482] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:31:11.017516 17849 raft_consensus.cc:3037] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:11.018257 17849 raft_consensus.cc:504] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a627ccc8f4473da4bf47168d982c15" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46205 } } peers { permanent_uuid: "57770afb759e454782060a6a1391037c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46467 } } peers { permanent_uuid: "744a758dc8934ef2a91799ae930d3b07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } }
I20260926 09:31:11.018394 17849 leader_election.cc:288] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07 [CANDIDATE]: Term 1 election: Requested vote from peers 32a627ccc8f4473da4bf47168d982c15 (127.0.98.1:46205), 57770afb759e454782060a6a1391037c (127.0.98.3:46467)
I20260926 09:31:11.018553 17513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28c94ba2a244e79a471d705a8fb722a" candidate_uuid: "744a758dc8934ef2a91799ae930d3b07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32a627ccc8f4473da4bf47168d982c15"
I20260926 09:31:11.018631 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28c94ba2a244e79a471d705a8fb722a" candidate_uuid: "744a758dc8934ef2a91799ae930d3b07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57770afb759e454782060a6a1391037c"
I20260926 09:31:11.018651 17513 raft_consensus.cc:3037] T a28c94ba2a244e79a471d705a8fb722a P 32a627ccc8f4473da4bf47168d982c15 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:11.018719 17781 raft_consensus.cc:3037] T a28c94ba2a244e79a471d705a8fb722a P 57770afb759e454782060a6a1391037c [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:11.019379 17781 raft_consensus.cc:2454] T a28c94ba2a244e79a471d705a8fb722a P 57770afb759e454782060a6a1391037c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744a758dc8934ef2a91799ae930d3b07 in term 1.
I20260926 09:31:11.019411 17513 raft_consensus.cc:2454] T a28c94ba2a244e79a471d705a8fb722a P 32a627ccc8f4473da4bf47168d982c15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 744a758dc8934ef2a91799ae930d3b07 in term 1.
I20260926 09:31:11.019601 17580 leader_election.cc:302] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32a627ccc8f4473da4bf47168d982c15, 744a758dc8934ef2a91799ae930d3b07; no voters:
I20260926 09:31:11.019704 17849 raft_consensus.cc:2781] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:11.019842 17849 raft_consensus.cc:686] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07 [term 1 LEADER]: Becoming Leader. State: Replica: 744a758dc8934ef2a91799ae930d3b07, State: Running, Role: LEADER
I20260926 09:31:11.019912 17849 consensus_queue.cc:237] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a627ccc8f4473da4bf47168d982c15" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46205 } } peers { permanent_uuid: "57770afb759e454782060a6a1391037c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46467 } } peers { permanent_uuid: "744a758dc8934ef2a91799ae930d3b07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } }
I20260926 09:31:11.020736 17368 catalog_manager.cc:5495] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 744a758dc8934ef2a91799ae930d3b07 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "744a758dc8934ef2a91799ae930d3b07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a627ccc8f4473da4bf47168d982c15" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57770afb759e454782060a6a1391037c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "744a758dc8934ef2a91799ae930d3b07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37865 } health_report { overall_health: HEALTHY } } }
W20260926 09:31:11.030740 17560 tablet.cc:2369] T a28c94ba2a244e79a471d705a8fb722a P 32a627ccc8f4473da4bf47168d982c15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:31:11.051031 17840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.048557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37810 (local address 127.0.98.2:37865)
0926 09:31:11.048617 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:11.048619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:11.048628 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:31:11.048674 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:11.048675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:11.048695 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:11.048759 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:11.048890 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.049360 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.050221 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.050414 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.050473 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:11.050665 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:11.050765 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:11.050835 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:11.050931 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 09:31:11.050951 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 09:31:11.053079 17781 raft_consensus.cc:1261] T a28c94ba2a244e79a471d705a8fb722a P 57770afb759e454782060a6a1391037c [term 1 FOLLOWER]: Refusing update from remote peer 744a758dc8934ef2a91799ae930d3b07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:31:11.053081 17513 raft_consensus.cc:1261] T a28c94ba2a244e79a471d705a8fb722a P 32a627ccc8f4473da4bf47168d982c15 [term 1 FOLLOWER]: Refusing update from remote peer 744a758dc8934ef2a91799ae930d3b07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:31:11.053503 17854 consensus_queue.cc:1035] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57770afb759e454782060a6a1391037c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:31:11.053669 17849 consensus_queue.cc:1035] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32a627ccc8f4473da4bf47168d982c15" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:31:11.056224 17858 mvcc.cc:205] Tried to move back new op lower bound from 7333540131031011328 to 7333540130898923520. Current Snapshot: MvccSnapshot[applied={T|T < 7333540131031011328}]
I20260926 09:31:11.057242 17859 mvcc.cc:205] Tried to move back new op lower bound from 7333540131031011328 to 7333540130898923520. Current Snapshot: MvccSnapshot[applied={T|T < 7333540131031011328 or (T in {7333540131031011328})}]
I20260926 09:31:11.058336 17368 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:31:11.058439 17368 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:31:11.059275 17368 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8aeac50451374948a366fdee1687fc8b: Sending DeleteTablet for 3 replicas of tablet a28c94ba2a244e79a471d705a8fb722a
I20260926 09:31:11.059636 17627 tablet_service.cc:1514] Processing DeleteTablet for tablet a28c94ba2a244e79a471d705a8fb722a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:31:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37804
I20260926 09:31:11.059765 17761 tablet_service.cc:1514] Processing DeleteTablet for tablet a28c94ba2a244e79a471d705a8fb722a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:31:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42028
I20260926 09:31:11.059762 17493 tablet_service.cc:1514] Processing DeleteTablet for tablet a28c94ba2a244e79a471d705a8fb722a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:31:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34482
I20260926 09:31:11.060112 17867 tablet_replica.cc:330] T a28c94ba2a244e79a471d705a8fb722a P 57770afb759e454782060a6a1391037c: stopping tablet replica
I20260926 09:31:11.060168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 17430
I20260926 09:31:11.060773 17867 raft_consensus.cc:2229] T a28c94ba2a244e79a471d705a8fb722a P 57770afb759e454782060a6a1391037c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:31:11.060892 17867 raft_consensus.cc:2258] T a28c94ba2a244e79a471d705a8fb722a P 57770afb759e454782060a6a1391037c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:31:11.061061 17866 tablet_replica.cc:330] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07: stopping tablet replica
I20260926 09:31:11.061151 17866 raft_consensus.cc:2229] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:31:11.061242 17866 raft_consensus.cc:2258] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:31:11.061683 17866 ts_tablet_manager.cc:1905] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:31:11.061684 17867 ts_tablet_manager.cc:1905] T a28c94ba2a244e79a471d705a8fb722a P 57770afb759e454782060a6a1391037c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:31:11.063395 17867 ts_tablet_manager.cc:1918] T a28c94ba2a244e79a471d705a8fb722a P 57770afb759e454782060a6a1391037c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:31:11.063475 17867 log.cc:1196] T a28c94ba2a244e79a471d705a8fb722a P 57770afb759e454782060a6a1391037c: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/a28c94ba2a244e79a471d705a8fb722a
I20260926 09:31:11.063413 17866 ts_tablet_manager.cc:1918] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:31:11.063633 17866 log.cc:1196] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414839069479-392-0/minicluster-data/ts-1/wal/wals/a28c94ba2a244e79a471d705a8fb722a
I20260926 09:31:11.063760 17867 ts_tablet_manager.cc:1939] T a28c94ba2a244e79a471d705a8fb722a P 57770afb759e454782060a6a1391037c: Deleting consensus metadata
I20260926 09:31:11.064049 17866 ts_tablet_manager.cc:1939] T a28c94ba2a244e79a471d705a8fb722a P 744a758dc8934ef2a91799ae930d3b07: Deleting consensus metadata
I20260926 09:31:11.064194 17352 catalog_manager.cc:4841] TS 57770afb759e454782060a6a1391037c (127.0.98.3:46467): tablet a28c94ba2a244e79a471d705a8fb722a (table test-table [id=0f25c6dd258845ffb43b716eb8dc4dd8]) successfully deleted
I20260926 09:31:11.064543 17354 catalog_manager.cc:4841] TS 744a758dc8934ef2a91799ae930d3b07 (127.0.98.2:37865): tablet a28c94ba2a244e79a471d705a8fb722a (table test-table [id=0f25c6dd258845ffb43b716eb8dc4dd8]) successfully deleted
W20260926 09:31:11.066754 17352 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46205 (error 108)
I20260926 09:31:11.066982 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 17564
I20260926 09:31:11.067965 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.067807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46205 (local address 127.0.0.1:34492)
0926 09:31:11.067876 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:11.067921 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 09:31:11.068117 17352 catalog_manager.cc:4584] TS 32a627ccc8f4473da4bf47168d982c15 (127.0.98.1:46205): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a28c94ba2a244e79a471d705a8fb722a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46205: connect: Connection refused (error 111)
I20260926 09:31:11.072898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 17698
I20260926 09:31:11.077180 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 17336
2026-09-26T09:31:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2099 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T09:31:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:11Z Disabled control of system clock
I20260926 09:31:11.105324 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37277
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 09:31:11.185796 17873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:11.186024 17873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:11.186044 17873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:11.187150 17873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:11.187186 17873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:11.187198 17873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:11.187211 17873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:11.187223 17873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:11.188485 17873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37277
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:11.188678 17873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:11.188959 17873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:11.193728 17878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:11.193750 17879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:11.193717 17881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:11.194047 17873 server_base.cc:1034] running on GCE node
I20260926 09:31:11.194306 17873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:11.194604 17873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:11.195766 17873 hybrid_clock.cc:648] HybridClock initialized: now 1790415071195744 us; error 40 us; skew 500 ppm
I20260926 09:31:11.197064 17873 webserver.cc:460] Webserver started at http://127.0.98.62:36007/ using document root <none> and password file <none>
I20260926 09:31:11.197321 17873 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:11.197369 17873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:11.197530 17873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:11.198642 17873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "76068e30d73d48119d892080cd084fce"
format_stamp: "Formatted at 2026-09-26 09:31:11 on dist-test-slave-dfzm"
server_key: "afced4ee6fffc5951aee28cded287770"
server_key_iv: "71df8088594ac35c8aae0d2644414a02"
server_key_version: "encryptionkey@0"
I20260926 09:31:11.199008 17873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "76068e30d73d48119d892080cd084fce"
format_stamp: "Formatted at 2026-09-26 09:31:11 on dist-test-slave-dfzm"
server_key: "afced4ee6fffc5951aee28cded287770"
server_key_iv: "71df8088594ac35c8aae0d2644414a02"
server_key_version: "encryptionkey@0"
I20260926 09:31:11.200301 17873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:31:11.201063 17887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:11.201237 17873 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:31:11.201336 17873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "76068e30d73d48119d892080cd084fce"
format_stamp: "Formatted at 2026-09-26 09:31:11 on dist-test-slave-dfzm"
server_key: "afced4ee6fffc5951aee28cded287770"
server_key_iv: "71df8088594ac35c8aae0d2644414a02"
server_key_version: "encryptionkey@0"
I20260926 09:31:11.201421 17873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:11.216749 17873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:11.217127 17873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:11.217291 17873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:11.221098 17873 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37277
I20260926 09:31:11.221148 17939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37277 every 8 connection(s)
I20260926 09:31:11.221524 17873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:11.222599 17940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:11.225040 17940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76068e30d73d48119d892080cd084fce: Bootstrap starting.
I20260926 09:31:11.225723 17940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76068e30d73d48119d892080cd084fce: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:11.226063 17940 log.cc:824] T 00000000000000000000000000000000 P 76068e30d73d48119d892080cd084fce: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:11.226845 17940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76068e30d73d48119d892080cd084fce: No bootstrap required, opened a new log
I20260926 09:31:11.228543 17940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76068e30d73d48119d892080cd084fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76068e30d73d48119d892080cd084fce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37277 } }
I20260926 09:31:11.228693 17940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76068e30d73d48119d892080cd084fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:11.228715 17940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76068e30d73d48119d892080cd084fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76068e30d73d48119d892080cd084fce, State: Initialized, Role: FOLLOWER
I20260926 09:31:11.228874 17940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76068e30d73d48119d892080cd084fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76068e30d73d48119d892080cd084fce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37277 } }
I20260926 09:31:11.228965 17940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76068e30d73d48119d892080cd084fce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:31:11.228991 17940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76068e30d73d48119d892080cd084fce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:31:11.229053 17940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76068e30d73d48119d892080cd084fce [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:11.229717 17940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76068e30d73d48119d892080cd084fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76068e30d73d48119d892080cd084fce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37277 } }
I20260926 09:31:11.229831 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 17873
I20260926 09:31:11.229815 17940 leader_election.cc:302] T 00000000000000000000000000000000 P 76068e30d73d48119d892080cd084fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76068e30d73d48119d892080cd084fce; no voters:
I20260926 09:31:11.229959 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:11.230079 17940 leader_election.cc:288] T 00000000000000000000000000000000 P 76068e30d73d48119d892080cd084fce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:31:11.230135 17943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76068e30d73d48119d892080cd084fce [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:11.230201 392 external_mini_cluster.cc:1428] Setting key 85e4fec445d5efbf30c402e7c7025d5a
I20260926 09:31:11.230340 17943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76068e30d73d48119d892080cd084fce [term 1 LEADER]: Becoming Leader. State: Replica: 76068e30d73d48119d892080cd084fce, State: Running, Role: LEADER
I20260926 09:31:11.230383 17940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76068e30d73d48119d892080cd084fce [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:11.230448 17943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76068e30d73d48119d892080cd084fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76068e30d73d48119d892080cd084fce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37277 } }
I20260926 09:31:11.231040 17945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76068e30d73d48119d892080cd084fce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76068e30d73d48119d892080cd084fce. Latest consensus state: current_term: 1 leader_uuid: "76068e30d73d48119d892080cd084fce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76068e30d73d48119d892080cd084fce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37277 } } }
I20260926 09:31:11.231170 17945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76068e30d73d48119d892080cd084fce [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:11.231793 17944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76068e30d73d48119d892080cd084fce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76068e30d73d48119d892080cd084fce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76068e30d73d48119d892080cd084fce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37277 } } }
I20260926 09:31:11.231940 17952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:11.232587 17952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:11.231951 17944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76068e30d73d48119d892080cd084fce [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:11.234261 17952 catalog_manager.cc:1347] Generated new cluster ID: 396059e1c4a84bdabbba48826af07964
I20260926 09:31:11.234320 17952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:31:11.236545 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.230711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34180 (local address 127.0.98.62:37277)
0926 09:31:11.231011 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:11.231021 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:11.231057 (+ 36us) server_negotiation.cc:407] Connection header received
0926 09:31:11.231261 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:11.231269 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:11.231349 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:11.231497 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:11.231831 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.233230 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.234292 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.234903 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.235021 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:11.235784 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:11.235788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:11.235788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:11.236012 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:11.236053 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:11.236067 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:11.236128 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:11.236233 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 09:31:11.236311 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":70,"threads_started":1}
I20260926 09:31:11.241513 17952 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:11.242238 17952 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:11.251076 17952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76068e30d73d48119d892080cd084fce: Generated new TSK 0
I20260926 09:31:11.251299 17952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:31:11.255218 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37277
--builtin_ntp_servers=127.0.98.20:45667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 09:31:11.336503 17964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:11.336717 17964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:11.336740 17964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:11.338002 17964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:11.338049 17964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:11.338078 17964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:31:11.339578 17964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:11.339838 17964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:11.340167 17964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:11.346276 17970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:11.346283 17969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:11.346351 17972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:11.346591 17964 server_base.cc:1034] running on GCE node
I20260926 09:31:11.346992 17964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:11.347295 17964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:11.348486 17964 hybrid_clock.cc:648] HybridClock initialized: now 1790415071348458 us; error 40 us; skew 500 ppm
I20260926 09:31:11.349974 17964 webserver.cc:460] Webserver started at http://127.0.98.1:43409/ using document root <none> and password file <none>
I20260926 09:31:11.350251 17964 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:11.350309 17964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:11.350438 17964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:11.351835 17964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "b761d56474434b148cbffdf8287e578d"
format_stamp: "Formatted at 2026-09-26 09:31:11 on dist-test-slave-dfzm"
server_key: "6b286e4162730cd5f1bd704fcc8fca3e"
server_key_iv: "4f863ea03df1f414fe53e30501227f6f"
server_key_version: "encryptionkey@0"
I20260926 09:31:11.352221 17964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b761d56474434b148cbffdf8287e578d"
format_stamp: "Formatted at 2026-09-26 09:31:11 on dist-test-slave-dfzm"
server_key: "6b286e4162730cd5f1bd704fcc8fca3e"
server_key_iv: "4f863ea03df1f414fe53e30501227f6f"
server_key_version: "encryptionkey@0"
I20260926 09:31:11.353950 17964 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260926 09:31:11.354920 17978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:11.355238 17964 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:31:11.355331 17964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "b761d56474434b148cbffdf8287e578d"
format_stamp: "Formatted at 2026-09-26 09:31:11 on dist-test-slave-dfzm"
server_key: "6b286e4162730cd5f1bd704fcc8fca3e"
server_key_iv: "4f863ea03df1f414fe53e30501227f6f"
server_key_version: "encryptionkey@0"
I20260926 09:31:11.355400 17964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:11.378480 17964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:11.378762 17964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:11.378868 17964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:11.379070 17964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:31:11.379448 17964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:31:11.379484 17964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:11.379541 17964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:31:11.379566 17964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:11.385491 17964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45863
I20260926 09:31:11.385541 18091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45863 every 8 connection(s)
I20260926 09:31:11.385933 17964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:31:11.390632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 17964
I20260926 09:31:11.390774 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:31:11.391016 392 external_mini_cluster.cc:1428] Setting key 4102446b485926ffdb975a65e6a5e014
I20260926 09:31:11.391537 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.386728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53363 (local address 127.0.98.62:37277)
0926 09:31:11.386776 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:11.386778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:11.387047 (+ 269us) server_negotiation.cc:407] Connection header received
0926 09:31:11.387608 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:11.387617 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:11.387632 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:11.387696 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:11.388388 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.388882 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.389944 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.390160 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.390232 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:11.391241 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:11.391243 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:11.391243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:11.391248 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:11.391263 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:11.391267 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:11.391327 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:11.391425 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 09:31:11.391460 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:31:11.391762 18094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.386712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37277 (local address 127.0.98.1:53363)
0926 09:31:11.386914 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:11.386948 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:11.387494 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:11.387769 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:11.387773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:11.387784 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:11.388291 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:11.388295 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.388980 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.388981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:11.389845 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:11.389848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.390250 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.390251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:11.390388 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:11.391091 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:31:11.391091 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:11.391132 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:31:11.391160 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:11.391327 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:11.391333 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:11.391336 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:31:11.391428 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 09:31:11.391531 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260926 09:31:11.392047 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37277
--builtin_ntp_servers=127.0.98.20:45667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 09:31:11.392403 18092 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37277
I20260926 09:31:11.392503 18092 heartbeater.cc:464] Registering TS with master...
I20260926 09:31:11.392702 18092 heartbeater.cc:511] Master 127.0.98.62:37277 requested a full tablet report, sending...
I20260926 09:31:11.393219 17904 ts_manager.cc:194] Registered new tserver with Master: b761d56474434b148cbffdf8287e578d (127.0.98.1:45863)
I20260926 09:31:11.394291 17904 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:53363
W20260926 09:31:11.474079 18095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:11.474289 18095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:11.474319 18095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:11.475410 18095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:11.475445 18095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:11.475472 18095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:31:11.476722 18095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:11.476969 18095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:11.477260 18095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:11.482370 18103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:11.482369 18101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:11.482391 18100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:11.482463 18095 server_base.cc:1034] running on GCE node
I20260926 09:31:11.482784 18095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:11.483034 18095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:11.484249 18095 hybrid_clock.cc:648] HybridClock initialized: now 1790415071484231 us; error 46 us; skew 500 ppm
I20260926 09:31:11.485545 18095 webserver.cc:460] Webserver started at http://127.0.98.2:35165/ using document root <none> and password file <none>
I20260926 09:31:11.485877 18095 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:11.485929 18095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:11.486057 18095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:11.487147 18095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "ddfd8f9b3f3a4c139a13e5102b55518e"
format_stamp: "Formatted at 2026-09-26 09:31:11 on dist-test-slave-dfzm"
server_key: "7962e93e89d14638e39cd9d41fccd863"
server_key_iv: "bedd7425720bdc0dcfe341e5d6590d83"
server_key_version: "encryptionkey@0"
I20260926 09:31:11.487504 18095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ddfd8f9b3f3a4c139a13e5102b55518e"
format_stamp: "Formatted at 2026-09-26 09:31:11 on dist-test-slave-dfzm"
server_key: "7962e93e89d14638e39cd9d41fccd863"
server_key_iv: "bedd7425720bdc0dcfe341e5d6590d83"
server_key_version: "encryptionkey@0"
I20260926 09:31:11.488819 18095 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:31:11.489705 18109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:11.489897 18095 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:11.489967 18095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "ddfd8f9b3f3a4c139a13e5102b55518e"
format_stamp: "Formatted at 2026-09-26 09:31:11 on dist-test-slave-dfzm"
server_key: "7962e93e89d14638e39cd9d41fccd863"
server_key_iv: "bedd7425720bdc0dcfe341e5d6590d83"
server_key_version: "encryptionkey@0"
I20260926 09:31:11.490073 18095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:11.505877 18095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:11.506191 18095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:11.506331 18095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:11.506569 18095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:31:11.506927 18095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:31:11.506968 18095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:11.506989 18095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:31:11.507002 18095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:11.512820 18095 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37741
I20260926 09:31:11.512908 18222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37741 every 8 connection(s)
I20260926 09:31:11.513270 18095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:31:11.517629 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 18095
I20260926 09:31:11.517763 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:31:11.517990 392 external_mini_cluster.cc:1428] Setting key 5348c314a3fb6c12c9b6f3fe35e6f249
I20260926 09:31:11.518644 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.514081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58563 (local address 127.0.98.62:37277)
0926 09:31:11.514134 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:11.514136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:11.514403 (+ 267us) server_negotiation.cc:407] Connection header received
0926 09:31:11.514868 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:11.514869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:11.514878 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:11.514932 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:11.515630 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.516111 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.517077 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.517300 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.517406 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:11.518401 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:11.518402 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:11.518402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:11.518406 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:11.518423 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:11.518427 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:11.518487 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:11.518537 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 09:31:11.518564 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:31:11.518918 18225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.514076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37277 (local address 127.0.98.2:58563)
0926 09:31:11.514273 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:11.514306 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:11.514781 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:11.514984 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:11.514987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:11.514995 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:11.515526 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:11.515530 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.516229 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.516230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:11.516989 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:11.516991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.517422 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.517423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:11.517538 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:11.518291 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:31:11.518291 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:11.518320 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:31:11.518342 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:11.518488 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:11.518490 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:11.518493 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:31:11.518537 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 09:31:11.518660 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":93,"threads_started":1}
I20260926 09:31:11.519059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37277
--builtin_ntp_servers=127.0.98.20:45667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 09:31:11.519425 18223 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37277
I20260926 09:31:11.519528 18223 heartbeater.cc:464] Registering TS with master...
I20260926 09:31:11.519704 18223 heartbeater.cc:511] Master 127.0.98.62:37277 requested a full tablet report, sending...
I20260926 09:31:11.520077 17904 ts_manager.cc:194] Registered new tserver with Master: ddfd8f9b3f3a4c139a13e5102b55518e (127.0.98.2:37741)
I20260926 09:31:11.520553 17904 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:58563
W20260926 09:31:11.601859 18226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:11.602067 18226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:11.602088 18226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:11.603276 18226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:11.603322 18226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:11.603348 18226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:31:11.604866 18226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:11.605136 18226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:11.605453 18226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:11.610520 18234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:11.610527 18232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:11.610535 18226 server_base.cc:1034] running on GCE node
W20260926 09:31:11.610518 18231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:11.611042 18226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:11.611311 18226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:11.612466 18226 hybrid_clock.cc:648] HybridClock initialized: now 1790415071612423 us; error 62 us; skew 500 ppm
I20260926 09:31:11.613865 18226 webserver.cc:460] Webserver started at http://127.0.98.3:44499/ using document root <none> and password file <none>
I20260926 09:31:11.614078 18226 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:11.614120 18226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:11.614220 18226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:11.615252 18226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "9ae0c5506b1b4f0b9ebd5a1de5c6d940"
format_stamp: "Formatted at 2026-09-26 09:31:11 on dist-test-slave-dfzm"
server_key: "5c84305e5a1e991c8e37f231efaca79b"
server_key_iv: "d023a1ade8e10688d415a3529c7ead96"
server_key_version: "encryptionkey@0"
I20260926 09:31:11.615572 18226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9ae0c5506b1b4f0b9ebd5a1de5c6d940"
format_stamp: "Formatted at 2026-09-26 09:31:11 on dist-test-slave-dfzm"
server_key: "5c84305e5a1e991c8e37f231efaca79b"
server_key_iv: "d023a1ade8e10688d415a3529c7ead96"
server_key_version: "encryptionkey@0"
I20260926 09:31:11.616927 18226 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:31:11.617841 18240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:11.618029 18226 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:11.618131 18226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "9ae0c5506b1b4f0b9ebd5a1de5c6d940"
format_stamp: "Formatted at 2026-09-26 09:31:11 on dist-test-slave-dfzm"
server_key: "5c84305e5a1e991c8e37f231efaca79b"
server_key_iv: "d023a1ade8e10688d415a3529c7ead96"
server_key_version: "encryptionkey@0"
I20260926 09:31:11.618216 18226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:11.634197 18226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:11.634542 18226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:11.634727 18226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:11.634979 18226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:31:11.635319 18226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:31:11.635383 18226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:11.635454 18226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:31:11.635485 18226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:11.641484 18226 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40039
I20260926 09:31:11.641533 18353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40039 every 8 connection(s)
I20260926 09:31:11.641983 18226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:31:11.644228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 18226
I20260926 09:31:11.644361 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:31:11.644593 392 external_mini_cluster.cc:1428] Setting key 76ae1a747034b336a41dd81bc5868db1
I20260926 09:31:11.647841 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.642821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59069 (local address 127.0.98.62:37277)
0926 09:31:11.642926 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:11.642929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:11.643139 (+ 210us) server_negotiation.cc:407] Connection header received
0926 09:31:11.643657 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:11.643658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:11.643669 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:11.643732 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:11.644509 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.645284 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.646381 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.646598 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.646662 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:11.647579 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:11.647580 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:11.647580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:11.647584 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:11.647598 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:11.647602 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:11.647657 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:11.647756 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 09:31:11.647781 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 09:31:11.648108 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.642741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37277 (local address 127.0.98.3:59069)
0926 09:31:11.642962 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:11.642997 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:11.643540 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:11.643807 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:11.643810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:11.643819 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:11.644409 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:11.644415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.645384 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.645385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:11.646282 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:11.646285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.646682 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.646682 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:11.646813 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:11.647430 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:31:11.647431 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:11.647462 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:31:11.647487 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:11.647668 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:11.647673 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:11.647676 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:31:11.647746 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 09:31:11.647830 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":83,"threads_started":1}
I20260926 09:31:11.648664 18354 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37277
I20260926 09:31:11.648751 18354 heartbeater.cc:464] Registering TS with master...
I20260926 09:31:11.648947 18354 heartbeater.cc:511] Master 127.0.98.62:37277 requested a full tablet report, sending...
I20260926 09:31:11.649470 17904 ts_manager.cc:194] Registered new tserver with Master: 9ae0c5506b1b4f0b9ebd5a1de5c6d940 (127.0.98.3:40039)
I20260926 09:31:11.649876 17904 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:59069
I20260926 09:31:11.656454 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:31:11.661139 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.658020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34192 (local address 127.0.98.62:37277)
0926 09:31:11.658085 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:11.658087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:11.658187 (+ 100us) server_negotiation.cc:407] Connection header received
0926 09:31:11.658240 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:11.658240 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:11.658250 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:11.658302 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:11.658518 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.659013 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.659877 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.660098 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.660167 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:11.660882 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:11.660883 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:11.660884 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:11.660888 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:11.660903 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:11.660907 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:11.660964 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:11.661022 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 09:31:11.661049 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:31:11.662588 17904 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:31:11.664160 17904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:31:11.670706 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.666994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52122 (local address 127.0.98.2:37741)
0926 09:31:11.667194 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:11.667199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:11.667208 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:31:11.667246 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:11.667248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:11.667264 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:11.667539 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:11.667597 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.668632 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.669778 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.670282 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.670383 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:11.670482 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:11.670569 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 09:31:11.670606 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":83,"threads_started":1}
I20260926 09:31:11.670692 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.666791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37741 (local address 127.0.0.1:52122)
0926 09:31:11.667007 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:11.667016 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:11.667055 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:11.667416 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:11.667416 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:11.667418 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:11.667462 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:11.667464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.668711 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.668712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:11.669687 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:11.669690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.670387 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.670388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:11.670463 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:11.670477 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 09:31:11.670554 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 09:31:11.670617 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":1,"thread_start_us":70,"threads_started":1}
I20260926 09:31:11.670991 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.666542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45863 (local address 127.0.0.1:50956)
0926 09:31:11.666682 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:11.666730 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:11.666801 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:11.667095 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:11.667097 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:11.667100 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:11.667171 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:11.667173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.668507 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.668508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:11.669709 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:11.669710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.670749 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.670749 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:11.670816 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:11.670820 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:31:11.670867 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 09:31:11.670907 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":8,"thread_start_us":74,"threads_started":1}
I20260926 09:31:11.671197 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.666542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40039 (local address 127.0.0.1:44764)
0926 09:31:11.666952 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:11.666962 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:11.667024 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:11.667413 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:11.667414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:11.667416 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:11.667474 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:11.667477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.668337 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.668338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:11.669708 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:11.669710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.670747 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.670748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:11.671034 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:11.671038 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:31:11.671091 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 09:31:11.671130 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":29,"thread_start_us":95,"threads_started":1}
I20260926 09:31:11.671227 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.666641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50956 (local address 127.0.98.1:45863)
0926 09:31:11.666870 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:11.666875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:11.666885 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:31:11.666923 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:11.666925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:11.666939 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:11.667014 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:11.667261 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.668416 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.669815 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.670551 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.670623 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:11.670718 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:11.671110 (+ 392us) server_negotiation.cc:299] Negotiation successful
0926 09:31:11.671150 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":99,"threads_started":1}
I20260926 09:31:11.671509 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.666745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44764 (local address 127.0.98.3:40039)
0926 09:31:11.667192 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:11.667196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:11.667208 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:31:11.667249 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:11.667251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:11.667267 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:11.667345 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:11.667551 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.668243 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.669815 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.670551 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.670623 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:11.670707 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:11.671359 (+ 652us) server_negotiation.cc:299] Negotiation successful
0926 09:31:11.671409 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":104,"threads_started":1}
I20260926 09:31:11.671610 18157 tablet_service.cc:1467] Processing CreateTablet for tablet f3304097b33742fda0aa826039b82da1 (DEFAULT_TABLE table=test-table [id=6421532c9e784412895b6e6e69d1b00d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:31:11.672008 18157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3304097b33742fda0aa826039b82da1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:11.672116 18288 tablet_service.cc:1467] Processing CreateTablet for tablet f3304097b33742fda0aa826039b82da1 (DEFAULT_TABLE table=test-table [id=6421532c9e784412895b6e6e69d1b00d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:31:11.672370 18288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3304097b33742fda0aa826039b82da1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:11.674388 18026 tablet_service.cc:1467] Processing CreateTablet for tablet f3304097b33742fda0aa826039b82da1 (DEFAULT_TABLE table=test-table [id=6421532c9e784412895b6e6e69d1b00d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:31:11.674717 18026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3304097b33742fda0aa826039b82da1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:11.674891 18368 tablet_bootstrap.cc:492] T f3304097b33742fda0aa826039b82da1 P 9ae0c5506b1b4f0b9ebd5a1de5c6d940: Bootstrap starting.
I20260926 09:31:11.675261 18369 tablet_bootstrap.cc:492] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e: Bootstrap starting.
I20260926 09:31:11.675482 18368 tablet_bootstrap.cc:654] T f3304097b33742fda0aa826039b82da1 P 9ae0c5506b1b4f0b9ebd5a1de5c6d940: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:11.675719 18368 log.cc:824] T f3304097b33742fda0aa826039b82da1 P 9ae0c5506b1b4f0b9ebd5a1de5c6d940: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:11.675902 18369 tablet_bootstrap.cc:654] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:11.676198 18369 log.cc:824] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:11.676345 18368 tablet_bootstrap.cc:492] T f3304097b33742fda0aa826039b82da1 P 9ae0c5506b1b4f0b9ebd5a1de5c6d940: No bootstrap required, opened a new log
I20260926 09:31:11.676411 18368 ts_tablet_manager.cc:1397] T f3304097b33742fda0aa826039b82da1 P 9ae0c5506b1b4f0b9ebd5a1de5c6d940: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:31:11.676941 18369 tablet_bootstrap.cc:492] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e: No bootstrap required, opened a new log
I20260926 09:31:11.676995 18369 ts_tablet_manager.cc:1397] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:31:11.678622 18368 raft_consensus.cc:348] T f3304097b33742fda0aa826039b82da1 P 9ae0c5506b1b4f0b9ebd5a1de5c6d940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae0c5506b1b4f0b9ebd5a1de5c6d940" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40039 } } peers { permanent_uuid: "b761d56474434b148cbffdf8287e578d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45863 } } peers { permanent_uuid: "ddfd8f9b3f3a4c139a13e5102b55518e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } }
I20260926 09:31:11.678798 18368 raft_consensus.cc:374] T f3304097b33742fda0aa826039b82da1 P 9ae0c5506b1b4f0b9ebd5a1de5c6d940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:11.678862 18368 raft_consensus.cc:729] T f3304097b33742fda0aa826039b82da1 P 9ae0c5506b1b4f0b9ebd5a1de5c6d940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae0c5506b1b4f0b9ebd5a1de5c6d940, State: Initialized, Role: FOLLOWER
I20260926 09:31:11.678987 18368 consensus_queue.cc:260] T f3304097b33742fda0aa826039b82da1 P 9ae0c5506b1b4f0b9ebd5a1de5c6d940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae0c5506b1b4f0b9ebd5a1de5c6d940" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40039 } } peers { permanent_uuid: "b761d56474434b148cbffdf8287e578d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45863 } } peers { permanent_uuid: "ddfd8f9b3f3a4c139a13e5102b55518e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } }
I20260926 09:31:11.679301 18368 ts_tablet_manager.cc:1428] T f3304097b33742fda0aa826039b82da1 P 9ae0c5506b1b4f0b9ebd5a1de5c6d940: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:31:11.679358 18354 heartbeater.cc:503] Master 127.0.98.62:37277 was elected leader, sending a full tablet report...
I20260926 09:31:11.679334 18369 raft_consensus.cc:348] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae0c5506b1b4f0b9ebd5a1de5c6d940" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40039 } } peers { permanent_uuid: "b761d56474434b148cbffdf8287e578d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45863 } } peers { permanent_uuid: "ddfd8f9b3f3a4c139a13e5102b55518e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } }
I20260926 09:31:11.679492 18369 raft_consensus.cc:374] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:11.679505 18372 tablet_bootstrap.cc:492] T f3304097b33742fda0aa826039b82da1 P b761d56474434b148cbffdf8287e578d: Bootstrap starting.
I20260926 09:31:11.679561 18369 raft_consensus.cc:729] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddfd8f9b3f3a4c139a13e5102b55518e, State: Initialized, Role: FOLLOWER
I20260926 09:31:11.679692 18369 consensus_queue.cc:260] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae0c5506b1b4f0b9ebd5a1de5c6d940" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40039 } } peers { permanent_uuid: "b761d56474434b148cbffdf8287e578d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45863 } } peers { permanent_uuid: "ddfd8f9b3f3a4c139a13e5102b55518e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } }
I20260926 09:31:11.679947 18369 ts_tablet_manager.cc:1428] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 09:31:11.679986 18223 heartbeater.cc:503] Master 127.0.98.62:37277 was elected leader, sending a full tablet report...
I20260926 09:31:11.680231 18372 tablet_bootstrap.cc:654] T f3304097b33742fda0aa826039b82da1 P b761d56474434b148cbffdf8287e578d: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:11.680555 18372 log.cc:824] T f3304097b33742fda0aa826039b82da1 P b761d56474434b148cbffdf8287e578d: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:11.681439 18372 tablet_bootstrap.cc:492] T f3304097b33742fda0aa826039b82da1 P b761d56474434b148cbffdf8287e578d: No bootstrap required, opened a new log
I20260926 09:31:11.681530 18372 ts_tablet_manager.cc:1397] T f3304097b33742fda0aa826039b82da1 P b761d56474434b148cbffdf8287e578d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:31:11.683275 18372 raft_consensus.cc:348] T f3304097b33742fda0aa826039b82da1 P b761d56474434b148cbffdf8287e578d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae0c5506b1b4f0b9ebd5a1de5c6d940" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40039 } } peers { permanent_uuid: "b761d56474434b148cbffdf8287e578d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45863 } } peers { permanent_uuid: "ddfd8f9b3f3a4c139a13e5102b55518e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } }
I20260926 09:31:11.683398 18372 raft_consensus.cc:374] T f3304097b33742fda0aa826039b82da1 P b761d56474434b148cbffdf8287e578d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:11.683418 18372 raft_consensus.cc:729] T f3304097b33742fda0aa826039b82da1 P b761d56474434b148cbffdf8287e578d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b761d56474434b148cbffdf8287e578d, State: Initialized, Role: FOLLOWER
I20260926 09:31:11.683531 18372 consensus_queue.cc:260] T f3304097b33742fda0aa826039b82da1 P b761d56474434b148cbffdf8287e578d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae0c5506b1b4f0b9ebd5a1de5c6d940" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40039 } } peers { permanent_uuid: "b761d56474434b148cbffdf8287e578d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45863 } } peers { permanent_uuid: "ddfd8f9b3f3a4c139a13e5102b55518e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } }
I20260926 09:31:11.683760 18372 ts_tablet_manager.cc:1428] T f3304097b33742fda0aa826039b82da1 P b761d56474434b148cbffdf8287e578d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 09:31:11.683849 18092 heartbeater.cc:503] Master 127.0.98.62:37277 was elected leader, sending a full tablet report...
W20260926 09:31:11.763872 18224 tablet.cc:2369] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:31:11.854573 18374 raft_consensus.cc:482] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:31:11.854705 18374 raft_consensus.cc:504] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae0c5506b1b4f0b9ebd5a1de5c6d940" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40039 } } peers { permanent_uuid: "b761d56474434b148cbffdf8287e578d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45863 } } peers { permanent_uuid: "ddfd8f9b3f3a4c139a13e5102b55518e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } }
I20260926 09:31:11.855149 18374 leader_election.cc:288] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ae0c5506b1b4f0b9ebd5a1de5c6d940 (127.0.98.3:40039), b761d56474434b148cbffdf8287e578d (127.0.98.1:45863)
I20260926 09:31:11.858268 18225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.855265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45863 (local address 127.0.98.2:34137)
0926 09:31:11.855335 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:11.855357 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:11.855403 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:11.855585 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:11.855587 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:11.855589 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:11.855696 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:11.855698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.856424 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.856425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:11.857307 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:11.857309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.858007 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.858008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:11.858077 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:11.858081 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:31:11.858140 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 09:31:11.858180 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 09:31:11.858327 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.855356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34137 (local address 127.0.98.1:45863)
0926 09:31:11.855410 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:11.855413 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:11.855428 (+ 15us) server_negotiation.cc:407] Connection header received
0926 09:31:11.855462 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:11.855463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:11.855477 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:11.855557 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:11.855788 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.856367 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.857403 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.857934 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.857991 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:11.858001 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:11.858192 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 09:31:11.858235 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 09:31:11.858631 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.855413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38169 (local address 127.0.98.3:40039)
0926 09:31:11.855457 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:11.855459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:11.855573 (+ 114us) server_negotiation.cc:407] Connection header received
0926 09:31:11.855674 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:11.855676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:11.855686 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:11.855752 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:11.855920 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.856451 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.857734 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.858215 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.858280 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:11.858290 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:11.858507 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 09:31:11.858552 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:31:11.858659 18046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3304097b33742fda0aa826039b82da1" candidate_uuid: "ddfd8f9b3f3a4c139a13e5102b55518e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b761d56474434b148cbffdf8287e578d" is_pre_election: true
I20260926 09:31:11.858831 18046 raft_consensus.cc:2454] T f3304097b33742fda0aa826039b82da1 P b761d56474434b148cbffdf8287e578d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddfd8f9b3f3a4c139a13e5102b55518e in term 0.
I20260926 09:31:11.858850 18377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.855305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40039 (local address 127.0.98.2:38169)
0926 09:31:11.855497 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:11.855508 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:11.855566 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:11.855781 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:11.855781 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:11.855783 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:11.855851 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:11.855853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.856537 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.856538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:11.857647 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:11.857651 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.858300 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.858301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:11.858395 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:11.858399 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:31:11.858725 (+ 326us) client_negotiation.cc:240] Negotiation successful
0926 09:31:11.858770 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":107,"threads_started":1}
I20260926 09:31:11.859068 18110 leader_election.cc:302] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b761d56474434b148cbffdf8287e578d, ddfd8f9b3f3a4c139a13e5102b55518e; no voters:
I20260926 09:31:11.859246 18374 raft_consensus.cc:2781] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:31:11.859234 18308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3304097b33742fda0aa826039b82da1" candidate_uuid: "ddfd8f9b3f3a4c139a13e5102b55518e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae0c5506b1b4f0b9ebd5a1de5c6d940" is_pre_election: true
I20260926 09:31:11.859318 18374 raft_consensus.cc:482] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:31:11.859345 18374 raft_consensus.cc:3037] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:11.859378 18308 raft_consensus.cc:2454] T f3304097b33742fda0aa826039b82da1 P 9ae0c5506b1b4f0b9ebd5a1de5c6d940 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddfd8f9b3f3a4c139a13e5102b55518e in term 0.
I20260926 09:31:11.860242 18374 raft_consensus.cc:504] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae0c5506b1b4f0b9ebd5a1de5c6d940" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40039 } } peers { permanent_uuid: "b761d56474434b148cbffdf8287e578d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45863 } } peers { permanent_uuid: "ddfd8f9b3f3a4c139a13e5102b55518e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } }
I20260926 09:31:11.860405 18374 leader_election.cc:288] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e [CANDIDATE]: Term 1 election: Requested vote from peers 9ae0c5506b1b4f0b9ebd5a1de5c6d940 (127.0.98.3:40039), b761d56474434b148cbffdf8287e578d (127.0.98.1:45863)
I20260926 09:31:11.860546 18308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3304097b33742fda0aa826039b82da1" candidate_uuid: "ddfd8f9b3f3a4c139a13e5102b55518e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae0c5506b1b4f0b9ebd5a1de5c6d940"
I20260926 09:31:11.860555 18046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3304097b33742fda0aa826039b82da1" candidate_uuid: "ddfd8f9b3f3a4c139a13e5102b55518e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b761d56474434b148cbffdf8287e578d"
I20260926 09:31:11.860666 18046 raft_consensus.cc:3037] T f3304097b33742fda0aa826039b82da1 P b761d56474434b148cbffdf8287e578d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:11.860666 18308 raft_consensus.cc:3037] T f3304097b33742fda0aa826039b82da1 P 9ae0c5506b1b4f0b9ebd5a1de5c6d940 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:11.861435 18046 raft_consensus.cc:2454] T f3304097b33742fda0aa826039b82da1 P b761d56474434b148cbffdf8287e578d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddfd8f9b3f3a4c139a13e5102b55518e in term 1.
I20260926 09:31:11.861509 18308 raft_consensus.cc:2454] T f3304097b33742fda0aa826039b82da1 P 9ae0c5506b1b4f0b9ebd5a1de5c6d940 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddfd8f9b3f3a4c139a13e5102b55518e in term 1.
I20260926 09:31:11.861663 18110 leader_election.cc:302] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b761d56474434b148cbffdf8287e578d, ddfd8f9b3f3a4c139a13e5102b55518e; no voters:
I20260926 09:31:11.861809 18374 raft_consensus.cc:2781] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:11.862053 18374 raft_consensus.cc:686] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e [term 1 LEADER]: Becoming Leader. State: Replica: ddfd8f9b3f3a4c139a13e5102b55518e, State: Running, Role: LEADER
I20260926 09:31:11.862169 18374 consensus_queue.cc:237] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae0c5506b1b4f0b9ebd5a1de5c6d940" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40039 } } peers { permanent_uuid: "b761d56474434b148cbffdf8287e578d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45863 } } peers { permanent_uuid: "ddfd8f9b3f3a4c139a13e5102b55518e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } }
I20260926 09:31:11.863054 17903 catalog_manager.cc:5495] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e reported cstate change: term changed from 0 to 1, leader changed from <none> to ddfd8f9b3f3a4c139a13e5102b55518e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ddfd8f9b3f3a4c139a13e5102b55518e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae0c5506b1b4f0b9ebd5a1de5c6d940" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b761d56474434b148cbffdf8287e578d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddfd8f9b3f3a4c139a13e5102b55518e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } health_report { overall_health: HEALTHY } } }
W20260926 09:31:11.886664 18093 tablet.cc:2369] T f3304097b33742fda0aa826039b82da1 P b761d56474434b148cbffdf8287e578d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:31:11.892611 18355 tablet.cc:2369] T f3304097b33742fda0aa826039b82da1 P 9ae0c5506b1b4f0b9ebd5a1de5c6d940: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:31:11.900810 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.898447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52136 (local address 127.0.98.2:37741)
0926 09:31:11.898514 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:11.898516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:11.898527 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:31:11.898593 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:11.898594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:11.898612 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:11.898675 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:11.898794 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.899260 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.900031 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:11.900261 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:11.900321 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:11.900458 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:11.900569 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:11.900629 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:11.900662 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 09:31:11.900677 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 09:31:11.902617 18046 raft_consensus.cc:1261] T f3304097b33742fda0aa826039b82da1 P b761d56474434b148cbffdf8287e578d [term 1 FOLLOWER]: Refusing update from remote peer ddfd8f9b3f3a4c139a13e5102b55518e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:31:11.902621 18308 raft_consensus.cc:1261] T f3304097b33742fda0aa826039b82da1 P 9ae0c5506b1b4f0b9ebd5a1de5c6d940 [term 1 FOLLOWER]: Refusing update from remote peer ddfd8f9b3f3a4c139a13e5102b55518e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:31:11.903028 18374 consensus_queue.cc:1035] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae0c5506b1b4f0b9ebd5a1de5c6d940" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:31:11.903198 18378 consensus_queue.cc:1035] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b761d56474434b148cbffdf8287e578d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:31:11.907848 17903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:31:11.907964 17903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:31:11.908730 17903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76068e30d73d48119d892080cd084fce: Sending DeleteTablet for 3 replicas of tablet f3304097b33742fda0aa826039b82da1
I20260926 09:31:11.909117 18026 tablet_service.cc:1514] Processing DeleteTablet for tablet f3304097b33742fda0aa826039b82da1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:31:11 UTC) from {username='slave'} at 127.0.0.1:50956
I20260926 09:31:11.909142 18157 tablet_service.cc:1514] Processing DeleteTablet for tablet f3304097b33742fda0aa826039b82da1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:31:11 UTC) from {username='slave'} at 127.0.0.1:52122
I20260926 09:31:11.909180 18288 tablet_service.cc:1514] Processing DeleteTablet for tablet f3304097b33742fda0aa826039b82da1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:31:11 UTC) from {username='slave'} at 127.0.0.1:44764
I20260926 09:31:11.909381 18392 tablet_replica.cc:330] T f3304097b33742fda0aa826039b82da1 P b761d56474434b148cbffdf8287e578d: stopping tablet replica
I20260926 09:31:11.909411 18393 tablet_replica.cc:330] T f3304097b33742fda0aa826039b82da1 P 9ae0c5506b1b4f0b9ebd5a1de5c6d940: stopping tablet replica
I20260926 09:31:11.909487 18392 raft_consensus.cc:2229] T f3304097b33742fda0aa826039b82da1 P b761d56474434b148cbffdf8287e578d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:31:11.909511 18393 raft_consensus.cc:2229] T f3304097b33742fda0aa826039b82da1 P 9ae0c5506b1b4f0b9ebd5a1de5c6d940 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:31:11.909636 18393 raft_consensus.cc:2258] T f3304097b33742fda0aa826039b82da1 P 9ae0c5506b1b4f0b9ebd5a1de5c6d940 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:31:11.909667 18392 raft_consensus.cc:2258] T f3304097b33742fda0aa826039b82da1 P b761d56474434b148cbffdf8287e578d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:31:11.909834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 17964
I20260926 09:31:11.910490 18393 ts_tablet_manager.cc:1905] T f3304097b33742fda0aa826039b82da1 P 9ae0c5506b1b4f0b9ebd5a1de5c6d940: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:31:11.912460 18393 ts_tablet_manager.cc:1918] T f3304097b33742fda0aa826039b82da1 P 9ae0c5506b1b4f0b9ebd5a1de5c6d940: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:31:11.912525 18393 log.cc:1196] T f3304097b33742fda0aa826039b82da1 P 9ae0c5506b1b4f0b9ebd5a1de5c6d940: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/f3304097b33742fda0aa826039b82da1
I20260926 09:31:11.912778 18393 ts_tablet_manager.cc:1939] T f3304097b33742fda0aa826039b82da1 P 9ae0c5506b1b4f0b9ebd5a1de5c6d940: Deleting consensus metadata
I20260926 09:31:11.913318 17891 catalog_manager.cc:4841] TS 9ae0c5506b1b4f0b9ebd5a1de5c6d940 (127.0.98.3:40039): tablet f3304097b33742fda0aa826039b82da1 (table test-table [id=6421532c9e784412895b6e6e69d1b00d]) successfully deleted
I20260926 09:31:11.914234 18391 tablet_replica.cc:330] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e: stopping tablet replica
I20260926 09:31:11.914361 18391 raft_consensus.cc:2229] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:31:11.914469 18391 raft_consensus.cc:2258] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:31:11.914836 18391 ts_tablet_manager.cc:1905] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:31:11.916270 18391 ts_tablet_manager.cc:1918] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:31:11.916358 18391 log.cc:1196] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414839069479-392-0/minicluster-data/ts-1/wal/wals/f3304097b33742fda0aa826039b82da1
I20260926 09:31:11.916616 18391 ts_tablet_manager.cc:1939] T f3304097b33742fda0aa826039b82da1 P ddfd8f9b3f3a4c139a13e5102b55518e: Deleting consensus metadata
I20260926 09:31:11.917011 17888 catalog_manager.cc:4841] TS ddfd8f9b3f3a4c139a13e5102b55518e (127.0.98.2:37741): tablet f3304097b33742fda0aa826039b82da1 (table test-table [id=6421532c9e784412895b6e6e69d1b00d]) successfully deleted
W20260926 09:31:11.917339 17888 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45863 (error 108)
I20260926 09:31:11.917552 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 18095
I20260926 09:31:11.918414 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:11.917707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45863 (local address 127.0.0.1:50972)
0926 09:31:11.918337 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:11.918386 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":604}
W20260926 09:31:11.918555 17888 catalog_manager.cc:4584] TS b761d56474434b148cbffdf8287e578d (127.0.98.1:45863): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f3304097b33742fda0aa826039b82da1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45863: connect: Connection refused (error 111)
I20260926 09:31:11.923878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 18226
I20260926 09:31:11.928411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 17873
2026-09-26T09:31:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (850 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:31:11 dist-test-slave-dfzm krb5kdc[18397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:31:11 dist-test-slave-dfzm krb5kdc[18397](info): set up 2 sockets
Sep 26 09:31:11 dist-test-slave-dfzm krb5kdc[18397](info): commencing operation
krb5kdc: starting...
W20260926 09:31:12.965428 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:31:13 dist-test-slave-dfzm krb5kdc[18397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:31:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:31:13.085938 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41129
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41129
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:13.181164 18410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:13.181371 18410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:13.181402 18410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:13.182561 18410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:13.182605 18410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:13.182618 18410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:13.182632 18410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:13.182644 18410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:13.184041 18410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:13.184221 18410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:13.184399 18410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:13.189157 18415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:13.189152 18416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:13.189219 18418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:13.189473 18410 server_base.cc:1034] running on GCE node
I20260926 09:31:13.189760 18410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:13.190059 18410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:13.191215 18410 hybrid_clock.cc:648] HybridClock initialized: now 1790415073191206 us; error 30 us; skew 500 ppm
Sep 26 09:31:13 dist-test-slave-dfzm krb5kdc[18397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:13.193781 18410 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:31:13.194281 18410 webserver.cc:460] Webserver started at http://127.0.98.62:45981/ using document root <none> and password file <none>
I20260926 09:31:13.194545 18410 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:13.194617 18410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:13.194772 18410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:13.195881 18410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "51b22eb7a18649e687ba7cdc87dfab23"
format_stamp: "Formatted at 2026-09-26 09:31:13 on dist-test-slave-dfzm"
server_key: "ce98e2785e5a88dac77feed8f9506302"
server_key_iv: "ff59c9e6079ae33609201cf8bc4a38e8"
server_key_version: "encryptionkey@0"
I20260926 09:31:13.196244 18410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "51b22eb7a18649e687ba7cdc87dfab23"
format_stamp: "Formatted at 2026-09-26 09:31:13 on dist-test-slave-dfzm"
server_key: "ce98e2785e5a88dac77feed8f9506302"
server_key_iv: "ff59c9e6079ae33609201cf8bc4a38e8"
server_key_version: "encryptionkey@0"
I20260926 09:31:13.197674 18410 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:31:13.198426 18425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:13.198700 18410 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:13.198820 18410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "51b22eb7a18649e687ba7cdc87dfab23"
format_stamp: "Formatted at 2026-09-26 09:31:13 on dist-test-slave-dfzm"
server_key: "ce98e2785e5a88dac77feed8f9506302"
server_key_iv: "ff59c9e6079ae33609201cf8bc4a38e8"
server_key_version: "encryptionkey@0"
I20260926 09:31:13.198906 18410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:13.238198 18410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:13.240828 18410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:13.241051 18410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:13.246031 18410 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41129
I20260926 09:31:13.246068 18477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41129 every 8 connection(s)
I20260926 09:31:13.246511 18410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:13.247488 18478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:13.250408 18478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51b22eb7a18649e687ba7cdc87dfab23: Bootstrap starting.
I20260926 09:31:13.250979 18478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51b22eb7a18649e687ba7cdc87dfab23: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:13.251231 18478 log.cc:824] T 00000000000000000000000000000000 P 51b22eb7a18649e687ba7cdc87dfab23: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:13.251979 18478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51b22eb7a18649e687ba7cdc87dfab23: No bootstrap required, opened a new log
I20260926 09:31:13.252063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 18410
I20260926 09:31:13.252195 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:13.252420 392 external_mini_cluster.cc:1428] Setting key e4b2c8527470a2f0ed55c4f2d37a4928
I20260926 09:31:13.254463 18478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51b22eb7a18649e687ba7cdc87dfab23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b22eb7a18649e687ba7cdc87dfab23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41129 } }
I20260926 09:31:13.254681 18478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51b22eb7a18649e687ba7cdc87dfab23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:13.254717 18478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51b22eb7a18649e687ba7cdc87dfab23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b22eb7a18649e687ba7cdc87dfab23, State: Initialized, Role: FOLLOWER
I20260926 09:31:13.254841 18478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51b22eb7a18649e687ba7cdc87dfab23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b22eb7a18649e687ba7cdc87dfab23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41129 } }
I20260926 09:31:13.254944 18478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51b22eb7a18649e687ba7cdc87dfab23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:31:13.254979 18478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51b22eb7a18649e687ba7cdc87dfab23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:31:13.255069 18478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51b22eb7a18649e687ba7cdc87dfab23 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:13.255667 18478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51b22eb7a18649e687ba7cdc87dfab23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b22eb7a18649e687ba7cdc87dfab23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41129 } }
I20260926 09:31:13.255774 18478 leader_election.cc:302] T 00000000000000000000000000000000 P 51b22eb7a18649e687ba7cdc87dfab23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51b22eb7a18649e687ba7cdc87dfab23; no voters:
I20260926 09:31:13.255957 18478 leader_election.cc:288] T 00000000000000000000000000000000 P 51b22eb7a18649e687ba7cdc87dfab23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:31:13.256070 18483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51b22eb7a18649e687ba7cdc87dfab23 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:13.256220 18478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51b22eb7a18649e687ba7cdc87dfab23 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:13.256238 18483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51b22eb7a18649e687ba7cdc87dfab23 [term 1 LEADER]: Becoming Leader. State: Replica: 51b22eb7a18649e687ba7cdc87dfab23, State: Running, Role: LEADER
I20260926 09:31:13.256495 18483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51b22eb7a18649e687ba7cdc87dfab23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b22eb7a18649e687ba7cdc87dfab23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41129 } }
I20260926 09:31:13.256872 18483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51b22eb7a18649e687ba7cdc87dfab23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51b22eb7a18649e687ba7cdc87dfab23. Latest consensus state: current_term: 1 leader_uuid: "51b22eb7a18649e687ba7cdc87dfab23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b22eb7a18649e687ba7cdc87dfab23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41129 } } }
I20260926 09:31:13.257112 18483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51b22eb7a18649e687ba7cdc87dfab23 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:13.257035 18485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51b22eb7a18649e687ba7cdc87dfab23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51b22eb7a18649e687ba7cdc87dfab23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b22eb7a18649e687ba7cdc87dfab23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41129 } } }
I20260926 09:31:13.257479 18485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51b22eb7a18649e687ba7cdc87dfab23 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:13.257731 18492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:13.258246 18492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:13.259960 18492 catalog_manager.cc:1347] Generated new cluster ID: dcf1fc6b578343b787a65e9f173a480c
I20260926 09:31:13.260015 18492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:31:13 dist-test-slave-dfzm krb5kdc[18397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:31:13.266677 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:13.252978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51798 (local address 127.0.98.62:41129)
0926 09:31:13.253194 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:13.253204 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:13.253239 (+ 35us) server_negotiation.cc:407] Connection header received
0926 09:31:13.254683 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:13.254692 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:13.254762 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:13.254941 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:13.255690 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.257328 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.258371 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.258878 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.259021 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:13.262635 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:13.262640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:13.262640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:13.262653 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:13.265018 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:13.265420 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:13.265423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:13.265424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:13.265510 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:13.265753 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:13.265754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:13.265754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:13.266023 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:13.266239 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:13.266320 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 09:31:13.266495 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":94,"threads_started":1}
I20260926 09:31:13.274441 18492 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:13.274973 18492 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:13.277967 18492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51b22eb7a18649e687ba7cdc87dfab23: Generated new TSK 0
I20260926 09:31:13.278153 18492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:31:13.308555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41129
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:13.390900 18504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:13.391140 18504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:13.391191 18504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:13.392360 18504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:13.392423 18504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:13.392470 18504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:31:13.394052 18504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:13.394331 18504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:13.394560 18504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:13.399500 18509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:13.399482 18510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:13.399668 18504 server_base.cc:1034] running on GCE node
W20260926 09:31:13.399500 18512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:13.399999 18504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:13.400249 18504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:13.401391 18504 hybrid_clock.cc:648] HybridClock initialized: now 1790415073401356 us; error 90 us; skew 500 ppm
Sep 26 09:31:13 dist-test-slave-dfzm krb5kdc[18397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:13.404129 18504 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:31:13.404592 18504 webserver.cc:460] Webserver started at http://127.0.98.1:44919/ using document root <none> and password file <none>
I20260926 09:31:13.404827 18504 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:13.404920 18504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:13.405066 18504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:13.406235 18504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "fe3dfa4a062545619e479c566737dac9"
format_stamp: "Formatted at 2026-09-26 09:31:13 on dist-test-slave-dfzm"
server_key: "82f3e06ab9f957b22c1bb92f7201f2d7"
server_key_iv: "43027235d0d5fb41f7d09ce9f2622c3a"
server_key_version: "encryptionkey@0"
I20260926 09:31:13.406610 18504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fe3dfa4a062545619e479c566737dac9"
format_stamp: "Formatted at 2026-09-26 09:31:13 on dist-test-slave-dfzm"
server_key: "82f3e06ab9f957b22c1bb92f7201f2d7"
server_key_iv: "43027235d0d5fb41f7d09ce9f2622c3a"
server_key_version: "encryptionkey@0"
I20260926 09:31:13.407941 18504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:31:13.408687 18519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:13.408871 18504 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:31:13.408987 18504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "fe3dfa4a062545619e479c566737dac9"
format_stamp: "Formatted at 2026-09-26 09:31:13 on dist-test-slave-dfzm"
server_key: "82f3e06ab9f957b22c1bb92f7201f2d7"
server_key_iv: "43027235d0d5fb41f7d09ce9f2622c3a"
server_key_version: "encryptionkey@0"
I20260926 09:31:13.409060 18504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:13.434928 18504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:13.437460 18504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:13.437685 18504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:13.437939 18504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:31:13.438265 18504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:31:13.438300 18504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:13.438323 18504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:31:13.438335 18504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:13.444283 18504 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43831
I20260926 09:31:13.444370 18632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43831 every 8 connection(s)
I20260926 09:31:13.444723 18504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 09:31:13 dist-test-slave-dfzm krb5kdc[18397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:31:13.454121 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 18504
I20260926 09:31:13.454276 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:31:13.454496 392 external_mini_cluster.cc:1428] Setting key a8d9ca4093d37d9806319305582bd8fd
I20260926 09:31:13.455253 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:13.445563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49549 (local address 127.0.98.62:41129)
0926 09:31:13.445691 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:13.445693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:13.445887 (+ 194us) server_negotiation.cc:407] Connection header received
0926 09:31:13.446931 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:13.446932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:13.446946 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:13.446999 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:13.447928 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.448451 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.449512 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.449797 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.449886 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:13.452158 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:13.452159 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:13.452160 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:13.452168 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:13.453724 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:13.454180 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:13.454181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:13.454182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:13.454245 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:13.454556 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:13.454557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:13.454557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:13.454690 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:13.454769 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:13.455062 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 09:31:13.455163 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 09:31:13.455459 18635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:13.445471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41129 (local address 127.0.98.1:49549)
0926 09:31:13.445730 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:13.445788 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:13.446809 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:13.447076 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:13.447080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:13.447583 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:13.447799 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:13.447805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.448551 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.448553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:13.449396 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:13.449399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.449898 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.449899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:13.450035 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:13.450606 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:13.450615 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:13.452045 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:13.453829 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:13.453837 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:13.453839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:13.454080 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:13.454315 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:13.454316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:13.454317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:13.454484 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:13.454760 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:13.454767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:13.454920 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 09:31:13.455029 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 09:31:13.455199 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":123,"threads_started":1}
I20260926 09:31:13.456168 18633 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41129
I20260926 09:31:13.456349 18633 heartbeater.cc:464] Registering TS with master...
I20260926 09:31:13.456564 18633 heartbeater.cc:511] Master 127.0.98.62:41129 requested a full tablet report, sending...
I20260926 09:31:13.457175 18442 ts_manager.cc:194] Registered new tserver with Master: fe3dfa4a062545619e479c566737dac9 (127.0.98.1:43831)
I20260926 09:31:13.458007 18442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49549
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:31:13.480908 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41129
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:13.560598 18638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:13.560813 18638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:13.560838 18638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:13.561973 18638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:13.562011 18638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:13.562036 18638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:31:13.563449 18638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:13.563632 18638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:13.563809 18638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:13.568763 18643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:13.568838 18646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:13.568773 18644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:13.568931 18638 server_base.cc:1034] running on GCE node
I20260926 09:31:13.569197 18638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:13.569406 18638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:13.570578 18638 hybrid_clock.cc:648] HybridClock initialized: now 1790415073570560 us; error 37 us; skew 500 ppm
Sep 26 09:31:13 dist-test-slave-dfzm krb5kdc[18397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:13.573110 18638 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:31:13.573632 18638 webserver.cc:460] Webserver started at http://127.0.98.2:34637/ using document root <none> and password file <none>
I20260926 09:31:13.573856 18638 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:13.573920 18638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:13.574069 18638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:13.575107 18638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "e4dcf28aa96548bdbb456d8b9f9f9281"
format_stamp: "Formatted at 2026-09-26 09:31:13 on dist-test-slave-dfzm"
server_key: "aa82bc413042711907901c029e152811"
server_key_iv: "f883e3863bd28ceacc68a121a6af894b"
server_key_version: "encryptionkey@0"
I20260926 09:31:13.575467 18638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e4dcf28aa96548bdbb456d8b9f9f9281"
format_stamp: "Formatted at 2026-09-26 09:31:13 on dist-test-slave-dfzm"
server_key: "aa82bc413042711907901c029e152811"
server_key_iv: "f883e3863bd28ceacc68a121a6af894b"
server_key_version: "encryptionkey@0"
I20260926 09:31:13.576792 18638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:31:13.577576 18653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:13.577819 18638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:13.577929 18638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "e4dcf28aa96548bdbb456d8b9f9f9281"
format_stamp: "Formatted at 2026-09-26 09:31:13 on dist-test-slave-dfzm"
server_key: "aa82bc413042711907901c029e152811"
server_key_iv: "f883e3863bd28ceacc68a121a6af894b"
server_key_version: "encryptionkey@0"
I20260926 09:31:13.578018 18638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:13.592741 18638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:13.595374 18638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:13.595551 18638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:13.595817 18638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:31:13.596202 18638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:31:13.596240 18638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:13.596267 18638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:31:13.596282 18638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:13.602792 18638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40265
I20260926 09:31:13.602869 18766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40265 every 8 connection(s)
I20260926 09:31:13.603195 18638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:31:13.605849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 18638
I20260926 09:31:13.605991 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:31:13.606246 392 external_mini_cluster.cc:1428] Setting key 80a8966b1a685b332dba3628b43f023b
Sep 26 09:31:13 dist-test-slave-dfzm krb5kdc[18397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:31:13.613870 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:13.603940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51725 (local address 127.0.98.62:41129)
0926 09:31:13.603987 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:13.603989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:13.604183 (+ 194us) server_negotiation.cc:407] Connection header received
0926 09:31:13.605260 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:13.605261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:13.605271 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:13.605326 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:13.606102 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.606830 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.607970 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.608197 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.608295 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:13.610688 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:13.610690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:13.610690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:13.610698 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:13.612369 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:13.612800 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:13.612802 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:13.612802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:13.612906 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:13.613163 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:13.613164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:13.613164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:13.613289 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:13.613370 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:13.613687 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 09:31:13.613769 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 09:31:13.614078 18769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:13.603923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41129 (local address 127.0.98.2:51725)
0926 09:31:13.604077 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:13.604113 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:13.605117 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:13.605390 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:13.605393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:13.605735 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:13.605954 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:13.605978 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.606942 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.606943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:13.607856 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:13.607859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.608309 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.608310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:13.608446 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:13.609026 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:13.609035 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:13.610588 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:13.612473 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:13.612480 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:13.612482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:13.612722 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:13.612989 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:13.612990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:13.612990 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:13.613095 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:13.613363 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:13.613370 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:13.613510 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 09:31:13.613665 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 09:31:13.613842 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260926 09:31:13.614669 18767 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41129
I20260926 09:31:13.614830 18767 heartbeater.cc:464] Registering TS with master...
I20260926 09:31:13.615078 18767 heartbeater.cc:511] Master 127.0.98.62:41129 requested a full tablet report, sending...
I20260926 09:31:13.615530 18442 ts_manager.cc:194] Registered new tserver with Master: e4dcf28aa96548bdbb456d8b9f9f9281 (127.0.98.2:40265)
I20260926 09:31:13.615928 18442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:31:13.632702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41129
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:13.713577 18772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:13.713845 18772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:13.713869 18772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:13.715240 18772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:13.715307 18772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:13.715350 18772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:31:13.717753 18772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:13.718017 18772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:13.718258 18772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:13.723098 18777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:13.723124 18778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:13.723132 18780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:13.723353 18772 server_base.cc:1034] running on GCE node
I20260926 09:31:13.723658 18772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:13.723879 18772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:13.725044 18772 hybrid_clock.cc:648] HybridClock initialized: now 1790415073725032 us; error 31 us; skew 500 ppm
Sep 26 09:31:13 dist-test-slave-dfzm krb5kdc[18397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:13.727640 18772 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:31:13.728107 18772 webserver.cc:460] Webserver started at http://127.0.98.3:46289/ using document root <none> and password file <none>
I20260926 09:31:13.728335 18772 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:13.728402 18772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:13.728551 18772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:13.729712 18772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "31bfe71d46cf4c7a9ca7a00b54b7f44b"
format_stamp: "Formatted at 2026-09-26 09:31:13 on dist-test-slave-dfzm"
server_key: "a5ed7d0ff6b5e424f1a19a5aa6b3c103"
server_key_iv: "9915faf8f08cb7868454f8d399222b36"
server_key_version: "encryptionkey@0"
I20260926 09:31:13.730078 18772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "31bfe71d46cf4c7a9ca7a00b54b7f44b"
format_stamp: "Formatted at 2026-09-26 09:31:13 on dist-test-slave-dfzm"
server_key: "a5ed7d0ff6b5e424f1a19a5aa6b3c103"
server_key_iv: "9915faf8f08cb7868454f8d399222b36"
server_key_version: "encryptionkey@0"
I20260926 09:31:13.731477 18772 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:31:13.732239 18787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:13.732410 18772 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:31:13.732514 18772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "31bfe71d46cf4c7a9ca7a00b54b7f44b"
format_stamp: "Formatted at 2026-09-26 09:31:13 on dist-test-slave-dfzm"
server_key: "a5ed7d0ff6b5e424f1a19a5aa6b3c103"
server_key_iv: "9915faf8f08cb7868454f8d399222b36"
server_key_version: "encryptionkey@0"
I20260926 09:31:13.732589 18772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:13.745460 18772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:13.747900 18772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:13.748064 18772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:13.748315 18772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:31:13.748677 18772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:31:13.748736 18772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:13.748772 18772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:31:13.748800 18772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:13.754976 18772 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36459
I20260926 09:31:13.755016 18900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36459 every 8 connection(s)
I20260926 09:31:13.755338 18772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:31:13.757720 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 18772
I20260926 09:31:13.757859 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:31:13.758035 392 external_mini_cluster.cc:1428] Setting key 8fc75725dc9fce0edb8bb0708c99eb29
Sep 26 09:31:13 dist-test-slave-dfzm krb5kdc[18397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:31:13.765219 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:13.756045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60605 (local address 127.0.98.62:41129)
0926 09:31:13.756099 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:13.756101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:13.756367 (+ 266us) server_negotiation.cc:407] Connection header received
0926 09:31:13.757516 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:13.757518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:13.757531 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:13.757615 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:13.758529 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.759170 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.760140 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.760363 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.760422 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:13.762546 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:13.762547 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:13.762548 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:13.762554 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:13.763740 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:13.764096 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:13.764097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:13.764097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:13.764147 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:13.764437 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:13.764438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:13.764439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:13.764597 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:13.764663 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:13.765004 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 09:31:13.765088 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:31:13.765241 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:13.756005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41129 (local address 127.0.98.3:60605)
0926 09:31:13.756220 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:13.756256 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:13.757368 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:13.757770 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:13.757774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:13.758173 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:13.758453 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:13.758459 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.759270 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.759271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:13.760062 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:13.760064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.760440 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.760441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:13.760536 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:13.761001 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:13.761009 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:13.762466 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:13.763814 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:13.763819 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:13.763821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:13.764030 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:13.764229 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:13.764230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:13.764231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:13.764347 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:13.764700 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:13.764705 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:13.764829 (+ 124us) client_negotiation.cc:769] Sending connection context
0926 09:31:13.764927 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 09:31:13.765050 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":92,"threads_started":1}
I20260926 09:31:13.765799 18901 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41129
I20260926 09:31:13.765919 18901 heartbeater.cc:464] Registering TS with master...
I20260926 09:31:13.766125 18901 heartbeater.cc:511] Master 127.0.98.62:41129 requested a full tablet report, sending...
I20260926 09:31:13.766542 18442 ts_manager.cc:194] Registered new tserver with Master: 31bfe71d46cf4c7a9ca7a00b54b7f44b (127.0.98.3:36459)
I20260926 09:31:13.766922 18442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60605
I20260926 09:31:13.769981 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:31:13.777112 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:13.771221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51800 (local address 127.0.98.62:41129)
0926 09:31:13.771289 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:13.771291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:13.771342 (+ 51us) server_negotiation.cc:407] Connection header received
0926 09:31:13.771387 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:13.771388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:13.771396 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:13.771441 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:13.771965 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.772438 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.773142 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.773324 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.773378 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:13.774759 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:13.774760 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:13.774760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:13.774766 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:13.775956 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:13.776325 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:13.776326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:13.776327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:13.776377 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:13.776558 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:13.776559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:13.776559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:13.776678 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:13.776766 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:13.776938 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 09:31:13.776991 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 09:31:13.778951 18442 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:31:13.780498 18442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:31:13.787150 18912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:13.782969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36459 (local address 127.0.0.1:55902)
0926 09:31:13.783258 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:13.783303 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:13.783384 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:13.783703 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:13.783706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:13.783709 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:13.783788 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:13.783790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.784773 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.784774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:13.785969 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:13.785971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.786833 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.786834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:13.786920 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:13.786935 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 09:31:13.787007 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 09:31:13.787055 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":66,"threads_started":1}
I20260926 09:31:13.787176 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:13.783143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55902 (local address 127.0.98.3:36459)
0926 09:31:13.783314 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:13.783319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:13.783358 (+ 39us) server_negotiation.cc:407] Connection header received
0926 09:31:13.783487 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:13.783495 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:13.783513 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:13.783598 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:13.783868 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.784669 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.786054 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.786743 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.786821 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:13.786881 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:13.787036 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 09:31:13.787087 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":70,"threads_started":1}
I20260926 09:31:13.787367 18913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:13.783228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43831 (local address 127.0.0.1:35258)
0926 09:31:13.783502 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:13.783512 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:13.783549 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:13.783847 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:13.783848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:13.783850 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:13.783899 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:13.783901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.785199 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.785200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:13.786242 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:13.786244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.787152 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.787153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:13.787221 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:13.787227 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 09:31:13.787260 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 09:31:13.787302 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":74,"threads_started":1}
I20260926 09:31:13.787688 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:13.783453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35258 (local address 127.0.98.1:43831)
0926 09:31:13.783659 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:13.783663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:13.783672 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:31:13.783714 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:13.783716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:13.783727 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:13.783978 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:13.784054 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.785096 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.786317 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.787043 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.787456 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:13.787506 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:13.787554 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 09:31:13.787598 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":103,"threads_started":1}
I20260926 09:31:13.788303 18567 tablet_service.cc:1467] Processing CreateTablet for tablet 0fd02a96e1324d32af26d72b698bc6bc (DEFAULT_TABLE table=test-table [id=1a445e07f5c3454eb9f8e95df349dc58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:31:13.788337 18835 tablet_service.cc:1467] Processing CreateTablet for tablet 0fd02a96e1324d32af26d72b698bc6bc (DEFAULT_TABLE table=test-table [id=1a445e07f5c3454eb9f8e95df349dc58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:31:13.788690 18835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fd02a96e1324d32af26d72b698bc6bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:13.788697 18567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fd02a96e1324d32af26d72b698bc6bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:13.788749 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:13.783062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47930 (local address 127.0.98.2:40265)
0926 09:31:13.783259 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:13.783265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:13.783417 (+ 152us) server_negotiation.cc:407] Connection header received
0926 09:31:13.783537 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:13.783540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:13.783554 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:13.783636 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:13.783930 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.785081 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.787192 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.788308 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.788357 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:13.788406 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:13.788623 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 09:31:13.788668 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":96,"threads_started":1}
I20260926 09:31:13.789068 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:13.782969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40265 (local address 127.0.0.1:47930)
0926 09:31:13.783346 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:13.783362 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:13.783447 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:13.783793 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:13.783794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:13.783797 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:13.783837 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:13.783838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.786311 (+ 2473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.786311 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:13.787087 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:13.787088 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.788467 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.788468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:13.788530 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:13.788533 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:31:13.788933 (+ 400us) client_negotiation.cc:240] Negotiation successful
0926 09:31:13.788978 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260926 09:31:13.789788 18701 tablet_service.cc:1467] Processing CreateTablet for tablet 0fd02a96e1324d32af26d72b698bc6bc (DEFAULT_TABLE table=test-table [id=1a445e07f5c3454eb9f8e95df349dc58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:31:13.790107 18701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fd02a96e1324d32af26d72b698bc6bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:13.791590 18918 tablet_bootstrap.cc:492] T 0fd02a96e1324d32af26d72b698bc6bc P fe3dfa4a062545619e479c566737dac9: Bootstrap starting.
I20260926 09:31:13.792027 18919 tablet_bootstrap.cc:492] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b: Bootstrap starting.
I20260926 09:31:13.792310 18918 tablet_bootstrap.cc:654] T 0fd02a96e1324d32af26d72b698bc6bc P fe3dfa4a062545619e479c566737dac9: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:13.792708 18919 tablet_bootstrap.cc:654] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:13.792876 18918 log.cc:824] T 0fd02a96e1324d32af26d72b698bc6bc P fe3dfa4a062545619e479c566737dac9: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:13.793272 18919 log.cc:824] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:13.793427 18921 tablet_bootstrap.cc:492] T 0fd02a96e1324d32af26d72b698bc6bc P e4dcf28aa96548bdbb456d8b9f9f9281: Bootstrap starting.
I20260926 09:31:13.793743 18918 tablet_bootstrap.cc:492] T 0fd02a96e1324d32af26d72b698bc6bc P fe3dfa4a062545619e479c566737dac9: No bootstrap required, opened a new log
I20260926 09:31:13.793870 18918 ts_tablet_manager.cc:1397] T 0fd02a96e1324d32af26d72b698bc6bc P fe3dfa4a062545619e479c566737dac9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:31:13.794104 18919 tablet_bootstrap.cc:492] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b: No bootstrap required, opened a new log
I20260926 09:31:13.794106 18921 tablet_bootstrap.cc:654] T 0fd02a96e1324d32af26d72b698bc6bc P e4dcf28aa96548bdbb456d8b9f9f9281: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:13.794190 18919 ts_tablet_manager.cc:1397] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:31:13.794697 18921 log.cc:824] T 0fd02a96e1324d32af26d72b698bc6bc P e4dcf28aa96548bdbb456d8b9f9f9281: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:13.795464 18921 tablet_bootstrap.cc:492] T 0fd02a96e1324d32af26d72b698bc6bc P e4dcf28aa96548bdbb456d8b9f9f9281: No bootstrap required, opened a new log
I20260926 09:31:13.795534 18921 ts_tablet_manager.cc:1397] T 0fd02a96e1324d32af26d72b698bc6bc P e4dcf28aa96548bdbb456d8b9f9f9281: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:31:13.795995 18919 raft_consensus.cc:348] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bfe71d46cf4c7a9ca7a00b54b7f44b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36459 } } peers { permanent_uuid: "fe3dfa4a062545619e479c566737dac9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43831 } } peers { permanent_uuid: "e4dcf28aa96548bdbb456d8b9f9f9281" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40265 } }
I20260926 09:31:13.796262 18919 raft_consensus.cc:374] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:13.796336 18919 raft_consensus.cc:729] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31bfe71d46cf4c7a9ca7a00b54b7f44b, State: Initialized, Role: FOLLOWER
I20260926 09:31:13.796300 18918 raft_consensus.cc:348] T 0fd02a96e1324d32af26d72b698bc6bc P fe3dfa4a062545619e479c566737dac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bfe71d46cf4c7a9ca7a00b54b7f44b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36459 } } peers { permanent_uuid: "fe3dfa4a062545619e479c566737dac9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43831 } } peers { permanent_uuid: "e4dcf28aa96548bdbb456d8b9f9f9281" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40265 } }
I20260926 09:31:13.796463 18919 consensus_queue.cc:260] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bfe71d46cf4c7a9ca7a00b54b7f44b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36459 } } peers { permanent_uuid: "fe3dfa4a062545619e479c566737dac9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43831 } } peers { permanent_uuid: "e4dcf28aa96548bdbb456d8b9f9f9281" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40265 } }
I20260926 09:31:13.796504 18918 raft_consensus.cc:374] T 0fd02a96e1324d32af26d72b698bc6bc P fe3dfa4a062545619e479c566737dac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:13.796550 18918 raft_consensus.cc:729] T 0fd02a96e1324d32af26d72b698bc6bc P fe3dfa4a062545619e479c566737dac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3dfa4a062545619e479c566737dac9, State: Initialized, Role: FOLLOWER
I20260926 09:31:13.796689 18918 consensus_queue.cc:260] T 0fd02a96e1324d32af26d72b698bc6bc P fe3dfa4a062545619e479c566737dac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bfe71d46cf4c7a9ca7a00b54b7f44b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36459 } } peers { permanent_uuid: "fe3dfa4a062545619e479c566737dac9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43831 } } peers { permanent_uuid: "e4dcf28aa96548bdbb456d8b9f9f9281" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40265 } }
I20260926 09:31:13.796751 18901 heartbeater.cc:503] Master 127.0.98.62:41129 was elected leader, sending a full tablet report...
I20260926 09:31:13.796995 18633 heartbeater.cc:503] Master 127.0.98.62:41129 was elected leader, sending a full tablet report...
I20260926 09:31:13.797067 18918 ts_tablet_manager.cc:1428] T 0fd02a96e1324d32af26d72b698bc6bc P fe3dfa4a062545619e479c566737dac9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:31:13.796751 18919 ts_tablet_manager.cc:1428] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:31:13.797569 18921 raft_consensus.cc:348] T 0fd02a96e1324d32af26d72b698bc6bc P e4dcf28aa96548bdbb456d8b9f9f9281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bfe71d46cf4c7a9ca7a00b54b7f44b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36459 } } peers { permanent_uuid: "fe3dfa4a062545619e479c566737dac9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43831 } } peers { permanent_uuid: "e4dcf28aa96548bdbb456d8b9f9f9281" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40265 } }
I20260926 09:31:13.797793 18921 raft_consensus.cc:374] T 0fd02a96e1324d32af26d72b698bc6bc P e4dcf28aa96548bdbb456d8b9f9f9281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:13.797843 18921 raft_consensus.cc:729] T 0fd02a96e1324d32af26d72b698bc6bc P e4dcf28aa96548bdbb456d8b9f9f9281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4dcf28aa96548bdbb456d8b9f9f9281, State: Initialized, Role: FOLLOWER
I20260926 09:31:13.797940 18921 consensus_queue.cc:260] T 0fd02a96e1324d32af26d72b698bc6bc P e4dcf28aa96548bdbb456d8b9f9f9281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bfe71d46cf4c7a9ca7a00b54b7f44b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36459 } } peers { permanent_uuid: "fe3dfa4a062545619e479c566737dac9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43831 } } peers { permanent_uuid: "e4dcf28aa96548bdbb456d8b9f9f9281" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40265 } }
I20260926 09:31:13.798205 18921 ts_tablet_manager.cc:1428] T 0fd02a96e1324d32af26d72b698bc6bc P e4dcf28aa96548bdbb456d8b9f9f9281: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 09:31:13.798230 18767 heartbeater.cc:503] Master 127.0.98.62:41129 was elected leader, sending a full tablet report...
W20260926 09:31:13.853821 18768 tablet.cc:2369] T 0fd02a96e1324d32af26d72b698bc6bc P e4dcf28aa96548bdbb456d8b9f9f9281: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:31:13.922907 18924 raft_consensus.cc:482] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:31:13.923056 18924 raft_consensus.cc:504] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bfe71d46cf4c7a9ca7a00b54b7f44b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36459 } } peers { permanent_uuid: "fe3dfa4a062545619e479c566737dac9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43831 } } peers { permanent_uuid: "e4dcf28aa96548bdbb456d8b9f9f9281" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40265 } }
I20260926 09:31:13.923472 18924 leader_election.cc:288] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe3dfa4a062545619e479c566737dac9 (127.0.98.1:43831), e4dcf28aa96548bdbb456d8b9f9f9281 (127.0.98.2:40265)
I20260926 09:31:13.927592 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:13.923746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45729 (local address 127.0.98.2:40265)
0926 09:31:13.923787 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:13.923789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:13.923978 (+ 189us) server_negotiation.cc:407] Connection header received
0926 09:31:13.924013 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:13.924014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:13.924025 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:13.924066 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:13.924263 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.924932 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.926601 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.927206 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.927271 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:13.927280 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:13.927475 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 09:31:13.927518 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:31:13.927610 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:13.923669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43831 (local address 127.0.98.3:51901)
0926 09:31:13.923726 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:13.923739 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:13.923783 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:13.923942 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:13.923943 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:13.923945 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:13.924018 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:13.924020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.925028 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.925028 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:13.926493 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:13.926496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.927299 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.927300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:13.927389 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:13.927392 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:31:13.927460 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 09:31:13.927503 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":130}
I20260926 09:31:13.927627 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:13.923674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40265 (local address 127.0.98.3:45729)
0926 09:31:13.923873 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:13.923881 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:13.923954 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:13.924144 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:13.924144 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:13.924146 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:13.924194 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:13.924195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.925027 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.925028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:13.926493 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:13.926496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.927299 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.927300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:13.927388 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:13.927392 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:31:13.927460 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 09:31:13.927503 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":53,"thread_start_us":118,"threads_started":1}
I20260926 09:31:13.927862 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:13.923751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51901 (local address 127.0.98.1:43831)
0926 09:31:13.923797 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:13.923800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:13.923809 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:31:13.923843 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:13.923844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:13.923856 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:13.923917 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:13.924121 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.924884 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.926601 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.927171 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.927254 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:13.927267 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:13.927751 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 09:31:13.927806 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:31:13.928099 18587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd02a96e1324d32af26d72b698bc6bc" candidate_uuid: "31bfe71d46cf4c7a9ca7a00b54b7f44b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3dfa4a062545619e479c566737dac9" is_pre_election: true
I20260926 09:31:13.928202 18721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd02a96e1324d32af26d72b698bc6bc" candidate_uuid: "31bfe71d46cf4c7a9ca7a00b54b7f44b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4dcf28aa96548bdbb456d8b9f9f9281" is_pre_election: true
I20260926 09:31:13.928305 18587 raft_consensus.cc:2454] T 0fd02a96e1324d32af26d72b698bc6bc P fe3dfa4a062545619e479c566737dac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31bfe71d46cf4c7a9ca7a00b54b7f44b in term 0.
I20260926 09:31:13.928421 18721 raft_consensus.cc:2454] T 0fd02a96e1324d32af26d72b698bc6bc P e4dcf28aa96548bdbb456d8b9f9f9281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31bfe71d46cf4c7a9ca7a00b54b7f44b in term 0.
I20260926 09:31:13.928572 18791 leader_election.cc:302] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31bfe71d46cf4c7a9ca7a00b54b7f44b, fe3dfa4a062545619e479c566737dac9; no voters:
I20260926 09:31:13.928719 18924 raft_consensus.cc:2781] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:31:13.928790 18924 raft_consensus.cc:482] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:31:13.928826 18924 raft_consensus.cc:3037] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:13.929513 18924 raft_consensus.cc:504] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bfe71d46cf4c7a9ca7a00b54b7f44b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36459 } } peers { permanent_uuid: "fe3dfa4a062545619e479c566737dac9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43831 } } peers { permanent_uuid: "e4dcf28aa96548bdbb456d8b9f9f9281" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40265 } }
I20260926 09:31:13.929704 18924 leader_election.cc:288] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b [CANDIDATE]: Term 1 election: Requested vote from peers fe3dfa4a062545619e479c566737dac9 (127.0.98.1:43831), e4dcf28aa96548bdbb456d8b9f9f9281 (127.0.98.2:40265)
I20260926 09:31:13.929836 18587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd02a96e1324d32af26d72b698bc6bc" candidate_uuid: "31bfe71d46cf4c7a9ca7a00b54b7f44b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3dfa4a062545619e479c566737dac9"
I20260926 09:31:13.929869 18721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd02a96e1324d32af26d72b698bc6bc" candidate_uuid: "31bfe71d46cf4c7a9ca7a00b54b7f44b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4dcf28aa96548bdbb456d8b9f9f9281"
I20260926 09:31:13.929925 18587 raft_consensus.cc:3037] T 0fd02a96e1324d32af26d72b698bc6bc P fe3dfa4a062545619e479c566737dac9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:13.929966 18721 raft_consensus.cc:3037] T 0fd02a96e1324d32af26d72b698bc6bc P e4dcf28aa96548bdbb456d8b9f9f9281 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:13.930577 18587 raft_consensus.cc:2454] T 0fd02a96e1324d32af26d72b698bc6bc P fe3dfa4a062545619e479c566737dac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31bfe71d46cf4c7a9ca7a00b54b7f44b in term 1.
I20260926 09:31:13.930732 18721 raft_consensus.cc:2454] T 0fd02a96e1324d32af26d72b698bc6bc P e4dcf28aa96548bdbb456d8b9f9f9281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31bfe71d46cf4c7a9ca7a00b54b7f44b in term 1.
I20260926 09:31:13.930744 18791 leader_election.cc:302] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31bfe71d46cf4c7a9ca7a00b54b7f44b, fe3dfa4a062545619e479c566737dac9; no voters:
I20260926 09:31:13.931053 18924 raft_consensus.cc:2781] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:13.931246 18924 raft_consensus.cc:686] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b [term 1 LEADER]: Becoming Leader. State: Replica: 31bfe71d46cf4c7a9ca7a00b54b7f44b, State: Running, Role: LEADER
I20260926 09:31:13.931371 18924 consensus_queue.cc:237] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bfe71d46cf4c7a9ca7a00b54b7f44b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36459 } } peers { permanent_uuid: "fe3dfa4a062545619e479c566737dac9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43831 } } peers { permanent_uuid: "e4dcf28aa96548bdbb456d8b9f9f9281" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40265 } }
I20260926 09:31:13.932210 18441 catalog_manager.cc:5495] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b reported cstate change: term changed from 0 to 1, leader changed from <none> to 31bfe71d46cf4c7a9ca7a00b54b7f44b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "31bfe71d46cf4c7a9ca7a00b54b7f44b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31bfe71d46cf4c7a9ca7a00b54b7f44b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3dfa4a062545619e479c566737dac9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4dcf28aa96548bdbb456d8b9f9f9281" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40265 } health_report { overall_health: UNKNOWN } } }
W20260926 09:31:13.945506 18634 tablet.cc:2369] T 0fd02a96e1324d32af26d72b698bc6bc P fe3dfa4a062545619e479c566737dac9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:31:13.968647 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:13.966394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55908 (local address 127.0.98.3:36459)
0926 09:31:13.966456 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:13.966458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:13.966466 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:31:13.966501 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:13.966502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:13.966520 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:13.966583 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:13.966691 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.967142 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.967834 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:13.968007 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:13.968055 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:13.968220 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:13.968303 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:13.968376 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:13.968509 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 09:31:13.968519 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 09:31:13.970908 18721 raft_consensus.cc:1261] T 0fd02a96e1324d32af26d72b698bc6bc P e4dcf28aa96548bdbb456d8b9f9f9281 [term 1 FOLLOWER]: Refusing update from remote peer 31bfe71d46cf4c7a9ca7a00b54b7f44b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:31:13.970988 18587 raft_consensus.cc:1261] T 0fd02a96e1324d32af26d72b698bc6bc P fe3dfa4a062545619e479c566737dac9 [term 1 FOLLOWER]: Refusing update from remote peer 31bfe71d46cf4c7a9ca7a00b54b7f44b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:31:13.971329 18928 consensus_queue.cc:1035] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4dcf28aa96548bdbb456d8b9f9f9281" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:31:13.971483 18924 consensus_queue.cc:1035] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3dfa4a062545619e479c566737dac9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:31:13.973726 18931 mvcc.cc:205] Tried to move back new op lower bound from 7333540142982623232 to 7333540142824034304. Current Snapshot: MvccSnapshot[applied={T|T < 7333540142982623232}]
I20260926 09:31:13.974419 18932 mvcc.cc:205] Tried to move back new op lower bound from 7333540142982623232 to 7333540142824034304. Current Snapshot: MvccSnapshot[applied={T|T < 7333540142982623232}]
I20260926 09:31:13.976601 18442 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:31:13.976698 18442 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:31:13.977454 18442 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51b22eb7a18649e687ba7cdc87dfab23: Sending DeleteTablet for 3 replicas of tablet 0fd02a96e1324d32af26d72b698bc6bc
I20260926 09:31:13.977792 18835 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fd02a96e1324d32af26d72b698bc6bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:31:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55902
I20260926 09:31:13.977859 18701 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fd02a96e1324d32af26d72b698bc6bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:31:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47930
I20260926 09:31:13.977850 18567 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fd02a96e1324d32af26d72b698bc6bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:31:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35258
I20260926 09:31:13.977999 18940 tablet_replica.cc:330] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b: stopping tablet replica
I20260926 09:31:13.978085 18940 raft_consensus.cc:2229] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:31:13.978209 18940 raft_consensus.cc:2258] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:31:13.978453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 18504
I20260926 09:31:13.979557 18940 ts_tablet_manager.cc:1905] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:31:13.981199 18940 ts_tablet_manager.cc:1918] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:31:13.981302 18940 log.cc:1196] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/0fd02a96e1324d32af26d72b698bc6bc
I20260926 09:31:13.981642 18940 ts_tablet_manager.cc:1939] T 0fd02a96e1324d32af26d72b698bc6bc P 31bfe71d46cf4c7a9ca7a00b54b7f44b: Deleting consensus metadata
I20260926 09:31:13.981721 18942 tablet_replica.cc:330] T 0fd02a96e1324d32af26d72b698bc6bc P e4dcf28aa96548bdbb456d8b9f9f9281: stopping tablet replica
I20260926 09:31:13.981837 18942 raft_consensus.cc:2229] T 0fd02a96e1324d32af26d72b698bc6bc P e4dcf28aa96548bdbb456d8b9f9f9281 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:31:13.981953 18942 raft_consensus.cc:2258] T 0fd02a96e1324d32af26d72b698bc6bc P e4dcf28aa96548bdbb456d8b9f9f9281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:31:13.982043 18429 catalog_manager.cc:4841] TS 31bfe71d46cf4c7a9ca7a00b54b7f44b (127.0.98.3:36459): tablet 0fd02a96e1324d32af26d72b698bc6bc (table test-table [id=1a445e07f5c3454eb9f8e95df349dc58]) successfully deleted
I20260926 09:31:13.982443 18942 ts_tablet_manager.cc:1905] T 0fd02a96e1324d32af26d72b698bc6bc P e4dcf28aa96548bdbb456d8b9f9f9281: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:31:13.984055 18942 ts_tablet_manager.cc:1918] T 0fd02a96e1324d32af26d72b698bc6bc P e4dcf28aa96548bdbb456d8b9f9f9281: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:31:13.984135 18942 log.cc:1196] T 0fd02a96e1324d32af26d72b698bc6bc P e4dcf28aa96548bdbb456d8b9f9f9281: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414839069479-392-0/minicluster-data/ts-1/wal/wals/0fd02a96e1324d32af26d72b698bc6bc
I20260926 09:31:13.984457 18942 ts_tablet_manager.cc:1939] T 0fd02a96e1324d32af26d72b698bc6bc P e4dcf28aa96548bdbb456d8b9f9f9281: Deleting consensus metadata
I20260926 09:31:13.984965 18426 catalog_manager.cc:4841] TS e4dcf28aa96548bdbb456d8b9f9f9281 (127.0.98.2:40265): tablet 0fd02a96e1324d32af26d72b698bc6bc (table test-table [id=1a445e07f5c3454eb9f8e95df349dc58]) successfully deleted
W20260926 09:31:13.985400 18429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43831 (error 108)
I20260926 09:31:13.985627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 18638
I20260926 09:31:13.986261 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:13.985795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43831 (local address 127.0.0.1:35270)
0926 09:31:13.986185 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:13.986220 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370}
W20260926 09:31:13.986660 18429 catalog_manager.cc:4584] TS fe3dfa4a062545619e479c566737dac9 (127.0.98.1:43831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0fd02a96e1324d32af26d72b698bc6bc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43831: connect: Connection refused (error 111)
I20260926 09:31:13.993139 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 18772
I20260926 09:31:13.998056 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 18410
2026-09-26T09:31:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2069 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T09:31:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:14Z Disabled control of system clock
I20260926 09:31:14.025341 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41293
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 09:31:14.105521 18947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:14.105782 18947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:14.105804 18947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:14.106927 18947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:14.106966 18947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:14.106979 18947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:14.106992 18947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:14.107003 18947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:14.108296 18947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41293
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:14.108508 18947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:14.108739 18947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:14.113727 18953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:14.113736 18955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:14.113765 18952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:14.113754 18947 server_base.cc:1034] running on GCE node
I20260926 09:31:14.114224 18947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:14.114497 18947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:14.115664 18947 hybrid_clock.cc:648] HybridClock initialized: now 1790415074115640 us; error 43 us; skew 500 ppm
I20260926 09:31:14.121968 18947 webserver.cc:460] Webserver started at http://127.0.98.62:33311/ using document root <none> and password file <none>
I20260926 09:31:14.122232 18947 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:14.122305 18947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:14.122457 18947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:14.123575 18947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "6fd12df3a7bf42a49f037656c48d8bb4"
format_stamp: "Formatted at 2026-09-26 09:31:14 on dist-test-slave-dfzm"
server_key: "31317333a2393742d3fba6d93763cd3b"
server_key_iv: "954c68d160481ff5ac6565ab967333fe"
server_key_version: "encryptionkey@0"
I20260926 09:31:14.123934 18947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "6fd12df3a7bf42a49f037656c48d8bb4"
format_stamp: "Formatted at 2026-09-26 09:31:14 on dist-test-slave-dfzm"
server_key: "31317333a2393742d3fba6d93763cd3b"
server_key_iv: "954c68d160481ff5ac6565ab967333fe"
server_key_version: "encryptionkey@0"
I20260926 09:31:14.125283 18947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:31:14.126143 18961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:14.126380 18947 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:31:14.126502 18947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "6fd12df3a7bf42a49f037656c48d8bb4"
format_stamp: "Formatted at 2026-09-26 09:31:14 on dist-test-slave-dfzm"
server_key: "31317333a2393742d3fba6d93763cd3b"
server_key_iv: "954c68d160481ff5ac6565ab967333fe"
server_key_version: "encryptionkey@0"
I20260926 09:31:14.126580 18947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:14.139139 18947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:14.139449 18947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:14.139601 18947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:14.143810 18947 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41293
I20260926 09:31:14.143853 19013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41293 every 8 connection(s)
I20260926 09:31:14.144232 18947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:14.145208 19014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:14.147614 19014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fd12df3a7bf42a49f037656c48d8bb4: Bootstrap starting.
I20260926 09:31:14.148224 19014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fd12df3a7bf42a49f037656c48d8bb4: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:14.148523 19014 log.cc:824] T 00000000000000000000000000000000 P 6fd12df3a7bf42a49f037656c48d8bb4: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:14.149299 19014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fd12df3a7bf42a49f037656c48d8bb4: No bootstrap required, opened a new log
I20260926 09:31:14.149755 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 18947
I20260926 09:31:14.149886 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:14.150127 392 external_mini_cluster.cc:1428] Setting key 1b1b591988131d68f9d18cf31d49e711
I20260926 09:31:14.151433 19014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fd12df3a7bf42a49f037656c48d8bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd12df3a7bf42a49f037656c48d8bb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41293 } }
I20260926 09:31:14.151602 19014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fd12df3a7bf42a49f037656c48d8bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:14.151640 19014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fd12df3a7bf42a49f037656c48d8bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fd12df3a7bf42a49f037656c48d8bb4, State: Initialized, Role: FOLLOWER
I20260926 09:31:14.151810 19014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fd12df3a7bf42a49f037656c48d8bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd12df3a7bf42a49f037656c48d8bb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41293 } }
I20260926 09:31:14.151909 19014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fd12df3a7bf42a49f037656c48d8bb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:31:14.151942 19014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fd12df3a7bf42a49f037656c48d8bb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:31:14.152015 19014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fd12df3a7bf42a49f037656c48d8bb4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:14.152559 19014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fd12df3a7bf42a49f037656c48d8bb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd12df3a7bf42a49f037656c48d8bb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41293 } }
I20260926 09:31:14.152676 19014 leader_election.cc:302] T 00000000000000000000000000000000 P 6fd12df3a7bf42a49f037656c48d8bb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fd12df3a7bf42a49f037656c48d8bb4; no voters:
I20260926 09:31:14.152878 19014 leader_election.cc:288] T 00000000000000000000000000000000 P 6fd12df3a7bf42a49f037656c48d8bb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:31:14.152911 19019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fd12df3a7bf42a49f037656c48d8bb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:14.153021 19019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fd12df3a7bf42a49f037656c48d8bb4 [term 1 LEADER]: Becoming Leader. State: Replica: 6fd12df3a7bf42a49f037656c48d8bb4, State: Running, Role: LEADER
I20260926 09:31:14.153163 19014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fd12df3a7bf42a49f037656c48d8bb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:14.153175 19019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fd12df3a7bf42a49f037656c48d8bb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd12df3a7bf42a49f037656c48d8bb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41293 } }
I20260926 09:31:14.153669 19020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fd12df3a7bf42a49f037656c48d8bb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fd12df3a7bf42a49f037656c48d8bb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd12df3a7bf42a49f037656c48d8bb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41293 } } }
I20260926 09:31:14.153934 19020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fd12df3a7bf42a49f037656c48d8bb4 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:14.153862 19021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fd12df3a7bf42a49f037656c48d8bb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fd12df3a7bf42a49f037656c48d8bb4. Latest consensus state: current_term: 1 leader_uuid: "6fd12df3a7bf42a49f037656c48d8bb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd12df3a7bf42a49f037656c48d8bb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41293 } } }
I20260926 09:31:14.154191 19021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fd12df3a7bf42a49f037656c48d8bb4 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:14.154345 19028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:14.154943 19028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:14.156741 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:14.150680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43374 (local address 127.0.98.62:41293)
0926 09:31:14.150892 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:14.150899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:14.150923 (+ 24us) server_negotiation.cc:407] Connection header received
0926 09:31:14.151419 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:14.151424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:14.151487 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:14.151621 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:14.151865 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.153318 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.154515 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.155086 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.155240 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:14.155993 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:14.155997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:14.155998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:14.156202 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:14.156244 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:14.156259 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:14.156328 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:14.156442 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 09:31:14.156528 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":93,"threads_started":1}
I20260926 09:31:14.157985 19028 catalog_manager.cc:1347] Generated new cluster ID: d570127bc6ce488eb2199e6158af7790
I20260926 09:31:14.158042 19028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:31:14.168708 19028 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:14.169421 19028 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:14.183180 19028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fd12df3a7bf42a49f037656c48d8bb4: Generated new TSK 0
I20260926 09:31:14.183384 19028 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:31:14.191696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41293
--builtin_ntp_servers=127.0.98.20:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 09:31:14.274283 19038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:14.274502 19038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:14.274524 19038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:14.275719 19038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:14.275765 19038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:14.275792 19038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:31:14.277200 19038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:14.277459 19038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:14.277794 19038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:14.285391 19046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:14.285391 19044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:14.285720 19043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:14.286098 19038 server_base.cc:1034] running on GCE node
I20260926 09:31:14.286345 19038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:14.286649 19038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:14.287801 19038 hybrid_clock.cc:648] HybridClock initialized: now 1790415074287681 us; error 136 us; skew 500 ppm
I20260926 09:31:14.289139 19038 webserver.cc:460] Webserver started at http://127.0.98.1:41837/ using document root <none> and password file <none>
I20260926 09:31:14.289387 19038 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:14.289455 19038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:14.289633 19038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:14.290723 19038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "75f08afaa9ab4a00851c8d695a7dd9bd"
format_stamp: "Formatted at 2026-09-26 09:31:14 on dist-test-slave-dfzm"
server_key: "1e1022478f7008261e11f17c205ebfd3"
server_key_iv: "b853ad30fc6e8d8df77580f8fb472f6b"
server_key_version: "encryptionkey@0"
I20260926 09:31:14.291095 19038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "75f08afaa9ab4a00851c8d695a7dd9bd"
format_stamp: "Formatted at 2026-09-26 09:31:14 on dist-test-slave-dfzm"
server_key: "1e1022478f7008261e11f17c205ebfd3"
server_key_iv: "b853ad30fc6e8d8df77580f8fb472f6b"
server_key_version: "encryptionkey@0"
I20260926 09:31:14.292397 19038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:31:14.293105 19052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:14.293286 19038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:31:14.293385 19038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "75f08afaa9ab4a00851c8d695a7dd9bd"
format_stamp: "Formatted at 2026-09-26 09:31:14 on dist-test-slave-dfzm"
server_key: "1e1022478f7008261e11f17c205ebfd3"
server_key_iv: "b853ad30fc6e8d8df77580f8fb472f6b"
server_key_version: "encryptionkey@0"
I20260926 09:31:14.293470 19038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:14.301885 19038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:14.302179 19038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:14.302325 19038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:14.302551 19038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:31:14.302865 19038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:31:14.302917 19038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:14.302958 19038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:31:14.302997 19038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:14.308977 19038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45951
I20260926 09:31:14.309029 19165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45951 every 8 connection(s)
I20260926 09:31:14.309348 19038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:31:14.314558 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:14.310113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46969 (local address 127.0.98.62:41293)
0926 09:31:14.310171 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:14.310173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:14.310453 (+ 280us) server_negotiation.cc:407] Connection header received
0926 09:31:14.310899 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:14.310900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:14.310912 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:14.310975 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:14.311641 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.312163 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.313135 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.313347 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.313425 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:14.314301 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:14.314302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:14.314303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:14.314307 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:14.314321 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:14.314325 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:14.314371 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:14.314471 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 09:31:14.314496 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:31:14.314704 19168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:14.310122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41293 (local address 127.0.98.1:46969)
0926 09:31:14.310309 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:14.310344 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:14.310812 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:14.311033 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:14.311036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:14.311043 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:14.311555 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:14.311560 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.312244 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.312245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:14.313041 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:14.313042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.313432 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.313433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:14.313544 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:14.314167 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:31:14.314167 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:14.314196 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:31:14.314217 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:14.314379 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:14.314384 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:14.314386 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:31:14.314439 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 09:31:14.314517 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260926 09:31:14.315189 19166 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41293
I20260926 09:31:14.315294 19166 heartbeater.cc:464] Registering TS with master...
I20260926 09:31:14.315478 19166 heartbeater.cc:511] Master 127.0.98.62:41293 requested a full tablet report, sending...
I20260926 09:31:14.315960 18978 ts_manager.cc:194] Registered new tserver with Master: 75f08afaa9ab4a00851c8d695a7dd9bd (127.0.98.1:45951)
I20260926 09:31:14.316526 18978 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:46969
I20260926 09:31:14.316766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 19038
I20260926 09:31:14.316886 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:31:14.317096 392 external_mini_cluster.cc:1428] Setting key 343a086da55a220c343bdb560a7495f9
I20260926 09:31:14.317929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41293
--builtin_ntp_servers=127.0.98.20:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 09:31:14.400952 19169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:14.401188 19169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:14.401227 19169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:14.402448 19169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:14.402511 19169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:14.402555 19169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:31:14.403904 19169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:14.404165 19169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:14.404470 19169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:14.409555 19175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:14.409662 19177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:14.409559 19174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:14.409934 19169 server_base.cc:1034] running on GCE node
I20260926 09:31:14.410109 19169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:14.410306 19169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:14.411451 19169 hybrid_clock.cc:648] HybridClock initialized: now 1790415074411441 us; error 27 us; skew 500 ppm
I20260926 09:31:14.412679 19169 webserver.cc:460] Webserver started at http://127.0.98.2:35693/ using document root <none> and password file <none>
I20260926 09:31:14.412914 19169 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:14.412959 19169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:14.413105 19169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:14.414254 19169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "9caf1d814b3b4e0a81b307d96a6e38c4"
format_stamp: "Formatted at 2026-09-26 09:31:14 on dist-test-slave-dfzm"
server_key: "93ce4ada4b84d9608fdeb61517457b0a"
server_key_iv: "4c24f4ed0e20ded368089b0e7e92df5a"
server_key_version: "encryptionkey@0"
I20260926 09:31:14.414618 19169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9caf1d814b3b4e0a81b307d96a6e38c4"
format_stamp: "Formatted at 2026-09-26 09:31:14 on dist-test-slave-dfzm"
server_key: "93ce4ada4b84d9608fdeb61517457b0a"
server_key_iv: "4c24f4ed0e20ded368089b0e7e92df5a"
server_key_version: "encryptionkey@0"
I20260926 09:31:14.415897 19169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:31:14.416646 19183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:14.416828 19169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:31:14.416900 19169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "9caf1d814b3b4e0a81b307d96a6e38c4"
format_stamp: "Formatted at 2026-09-26 09:31:14 on dist-test-slave-dfzm"
server_key: "93ce4ada4b84d9608fdeb61517457b0a"
server_key_iv: "4c24f4ed0e20ded368089b0e7e92df5a"
server_key_version: "encryptionkey@0"
I20260926 09:31:14.416988 19169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:14.441934 19169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:14.442304 19169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:14.442487 19169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:14.442756 19169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:31:14.443118 19169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:31:14.443174 19169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:14.443216 19169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:31:14.443244 19169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:14.449046 19169 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33177
I20260926 09:31:14.449087 19296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33177 every 8 connection(s)
I20260926 09:31:14.449426 19169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:31:14.452829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 19169
I20260926 09:31:14.452955 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:31:14.453243 392 external_mini_cluster.cc:1428] Setting key b9e460f061aef34aa5f49c3f3d6f5120
I20260926 09:31:14.454339 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41293
--builtin_ntp_servers=127.0.98.20:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 09:31:14.455170 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:14.450177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58449 (local address 127.0.98.62:41293)
0926 09:31:14.450266 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:14.450269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:14.450482 (+ 213us) server_negotiation.cc:407] Connection header received
0926 09:31:14.450965 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:14.450966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:14.450977 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:14.451056 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:14.451752 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.452251 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.453525 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.453811 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.453887 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:14.454899 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:14.454901 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:14.454901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:14.454907 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:14.454922 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:14.454927 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:14.454972 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:14.455060 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 09:31:14.455091 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 09:31:14.455410 19299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:14.450136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41293 (local address 127.0.98.2:58449)
0926 09:31:14.450336 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:14.450395 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:14.450870 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:14.451132 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:14.451136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:14.451145 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:14.451665 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:14.451670 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.452341 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.452341 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:14.453382 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:14.453385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.453881 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.453882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:14.454061 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:14.454744 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:31:14.454745 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:14.454775 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:31:14.454799 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:14.454983 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:14.454987 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:14.454990 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:31:14.455062 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 09:31:14.455169 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":104,"threads_started":1}
I20260926 09:31:14.456027 19297 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41293
I20260926 09:31:14.456125 19297 heartbeater.cc:464] Registering TS with master...
I20260926 09:31:14.456327 19297 heartbeater.cc:511] Master 127.0.98.62:41293 requested a full tablet report, sending...
I20260926 09:31:14.456700 18978 ts_manager.cc:194] Registered new tserver with Master: 9caf1d814b3b4e0a81b307d96a6e38c4 (127.0.98.2:33177)
I20260926 09:31:14.457083 18978 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:58449
W20260926 09:31:14.536005 19300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:14.536201 19300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:14.536221 19300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:14.537372 19300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:14.537415 19300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:14.537438 19300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:31:14.538790 19300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:14.539034 19300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:14.539284 19300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:14.545145 19308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:14.545162 19305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:14.545154 19306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:14.545472 19300 server_base.cc:1034] running on GCE node
I20260926 09:31:14.545719 19300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:14.545984 19300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:14.547125 19300 hybrid_clock.cc:648] HybridClock initialized: now 1790415074547105 us; error 36 us; skew 500 ppm
I20260926 09:31:14.548360 19300 webserver.cc:460] Webserver started at http://127.0.98.3:38641/ using document root <none> and password file <none>
I20260926 09:31:14.548624 19300 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:14.548671 19300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:14.548823 19300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:14.550000 19300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "8a284ede62614acfbb33e00a6291ed54"
format_stamp: "Formatted at 2026-09-26 09:31:14 on dist-test-slave-dfzm"
server_key: "ef69cf540ed179da5ff9b57ea0f1e1f7"
server_key_iv: "20ce43cf5ad78fc477c6781a5cbe3101"
server_key_version: "encryptionkey@0"
I20260926 09:31:14.550354 19300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8a284ede62614acfbb33e00a6291ed54"
format_stamp: "Formatted at 2026-09-26 09:31:14 on dist-test-slave-dfzm"
server_key: "ef69cf540ed179da5ff9b57ea0f1e1f7"
server_key_iv: "20ce43cf5ad78fc477c6781a5cbe3101"
server_key_version: "encryptionkey@0"
I20260926 09:31:14.551610 19300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:31:14.552296 19314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:14.552459 19300 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:31:14.552547 19300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "8a284ede62614acfbb33e00a6291ed54"
format_stamp: "Formatted at 2026-09-26 09:31:14 on dist-test-slave-dfzm"
server_key: "ef69cf540ed179da5ff9b57ea0f1e1f7"
server_key_iv: "20ce43cf5ad78fc477c6781a5cbe3101"
server_key_version: "encryptionkey@0"
I20260926 09:31:14.552620 19300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:14.571017 19300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:14.571295 19300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:14.571408 19300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:14.571601 19300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:31:14.571954 19300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:31:14.571987 19300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:14.572010 19300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:31:14.572058 19300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:14.578173 19300 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32925
I20260926 09:31:14.578279 19427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32925 every 8 connection(s)
I20260926 09:31:14.578612 19300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:31:14.579558 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 19300
I20260926 09:31:14.579679 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:31:14.579862 392 external_mini_cluster.cc:1428] Setting key c543e57e24fb53f075d39f548adbcbdd
I20260926 09:31:14.583954 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:14.579399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54053 (local address 127.0.98.62:41293)
0926 09:31:14.579453 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:14.579456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:14.579693 (+ 237us) server_negotiation.cc:407] Connection header received
0926 09:31:14.580287 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:14.580288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:14.580300 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:14.580343 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:14.581017 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.581522 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.582570 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.582770 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.582880 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:14.583666 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:14.583667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:14.583668 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:14.583674 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:14.583688 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:14.583693 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:14.583736 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:14.583845 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 09:31:14.583876 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:31:14.584170 19430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:14.579366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41293 (local address 127.0.98.3:54053)
0926 09:31:14.579567 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:14.579608 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:14.580182 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:14.580423 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:14.580427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:14.580438 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:14.580928 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:14.580932 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.581650 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.581651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:14.582472 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:14.582475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.582888 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.582890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:14.583032 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:14.583528 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:31:14.583528 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:14.583556 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:31:14.583579 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:14.583747 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:14.583752 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:14.583755 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:31:14.583840 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 09:31:14.583940 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":95,"threads_started":1}
I20260926 09:31:14.584712 19428 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41293
I20260926 09:31:14.584800 19428 heartbeater.cc:464] Registering TS with master...
I20260926 09:31:14.584944 19428 heartbeater.cc:511] Master 127.0.98.62:41293 requested a full tablet report, sending...
I20260926 09:31:14.585309 18978 ts_manager.cc:194] Registered new tserver with Master: 8a284ede62614acfbb33e00a6291ed54 (127.0.98.3:32925)
I20260926 09:31:14.585669 18978 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:54053
I20260926 09:31:14.591591 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:31:14.595811 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:14.593004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42790 (local address 127.0.98.62:41293)
0926 09:31:14.593047 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:14.593049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:14.593115 (+ 66us) server_negotiation.cc:407] Connection header received
0926 09:31:14.593167 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:14.593168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:14.593177 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:14.593226 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:14.593359 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.593845 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.594650 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.594827 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.594881 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:14.595562 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:14.595564 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:14.595564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:14.595568 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:14.595581 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:14.595585 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:14.595661 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:14.595745 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 09:31:14.595755 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:31:14.597751 18978 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:31:14.599265 18978 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:31:14.605783 19440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:14.601655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33177 (local address 127.0.0.1:45826)
0926 09:31:14.601915 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:14.601924 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:14.601965 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:14.602347 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:14.602348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:14.602350 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:14.602399 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:14.602401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.603400 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.603401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:14.604661 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:14.604664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.605428 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.605429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:14.605515 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:14.605529 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 09:31:14.605642 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 09:31:14.605701 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":76,"threads_started":1}
I20260926 09:31:14.605798 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:14.601900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45826 (local address 127.0.98.2:33177)
0926 09:31:14.602084 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:14.602089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:14.602101 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:31:14.602149 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:14.602150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:14.602185 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:14.602275 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:14.602489 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.603292 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.604779 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.605320 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.605426 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:14.605536 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:14.605668 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 09:31:14.605716 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":115,"threads_started":1}
I20260926 09:31:14.606011 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:14.601443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45951 (local address 127.0.0.1:50554)
0926 09:31:14.601628 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:14.601679 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:14.601759 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:14.602538 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:14.602539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:14.602540 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:14.602573 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:14.602574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.603770 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.603771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:14.604808 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:14.604810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.605792 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.605793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:14.605847 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:14.605849 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:31:14.605896 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 09:31:14.605933 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":56,"threads_started":1}
I20260926 09:31:14.606184 19439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:14.601636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39560 (local address 127.0.98.3:32925)
0926 09:31:14.602071 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:14.602075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:14.602118 (+ 43us) server_negotiation.cc:407] Connection header received
0926 09:31:14.602159 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:14.602161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:14.602178 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:14.602264 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:14.602458 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.603599 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.604705 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.605440 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.605527 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:14.605639 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:14.606064 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 09:31:14.606107 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":95,"threads_started":1}
I20260926 09:31:14.606191 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:14.601554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50554 (local address 127.0.98.1:45951)
0926 09:31:14.602395 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:14.602400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:14.602409 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:31:14.602447 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:14.602449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:14.602463 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:14.602538 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:14.602650 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.603687 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.604892 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.605621 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.605683 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:14.605760 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:14.606076 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 09:31:14.606114 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":799,"thread_start_us":131,"threads_started":1}
I20260926 09:31:14.606321 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:14.601444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32925 (local address 127.0.0.1:39560)
0926 09:31:14.601832 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:14.601843 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:14.601902 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:14.602293 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:14.602296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:14.602298 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:14.602367 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:14.602369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.603699 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.603701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:14.604620 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:14.604622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.605675 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.605676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:14.605735 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:14.605739 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:31:14.605804 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 09:31:14.605845 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":6,"thread_start_us":92,"threads_started":1}
I20260926 09:31:14.606968 19231 tablet_service.cc:1467] Processing CreateTablet for tablet 87f489b6b4e744c49e78570ba90508d5 (DEFAULT_TABLE table=test-table [id=e68aa47995aa409c8fa8af152ec4bb40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:31:14.607371 19231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87f489b6b4e744c49e78570ba90508d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:14.607472 19362 tablet_service.cc:1467] Processing CreateTablet for tablet 87f489b6b4e744c49e78570ba90508d5 (DEFAULT_TABLE table=test-table [id=e68aa47995aa409c8fa8af152ec4bb40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:31:14.607797 19362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87f489b6b4e744c49e78570ba90508d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:14.610292 19442 tablet_bootstrap.cc:492] T 87f489b6b4e744c49e78570ba90508d5 P 9caf1d814b3b4e0a81b307d96a6e38c4: Bootstrap starting.
I20260926 09:31:14.610735 19443 tablet_bootstrap.cc:492] T 87f489b6b4e744c49e78570ba90508d5 P 8a284ede62614acfbb33e00a6291ed54: Bootstrap starting.
I20260926 09:31:14.610697 19100 tablet_service.cc:1467] Processing CreateTablet for tablet 87f489b6b4e744c49e78570ba90508d5 (DEFAULT_TABLE table=test-table [id=e68aa47995aa409c8fa8af152ec4bb40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:31:14.611060 19100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87f489b6b4e744c49e78570ba90508d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:14.611075 19442 tablet_bootstrap.cc:654] T 87f489b6b4e744c49e78570ba90508d5 P 9caf1d814b3b4e0a81b307d96a6e38c4: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:14.611387 19443 tablet_bootstrap.cc:654] T 87f489b6b4e744c49e78570ba90508d5 P 8a284ede62614acfbb33e00a6291ed54: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:14.611411 19442 log.cc:824] T 87f489b6b4e744c49e78570ba90508d5 P 9caf1d814b3b4e0a81b307d96a6e38c4: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:14.611703 19443 log.cc:824] T 87f489b6b4e744c49e78570ba90508d5 P 8a284ede62614acfbb33e00a6291ed54: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:14.612215 19442 tablet_bootstrap.cc:492] T 87f489b6b4e744c49e78570ba90508d5 P 9caf1d814b3b4e0a81b307d96a6e38c4: No bootstrap required, opened a new log
I20260926 09:31:14.612296 19442 ts_tablet_manager.cc:1397] T 87f489b6b4e744c49e78570ba90508d5 P 9caf1d814b3b4e0a81b307d96a6e38c4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:31:14.612476 19443 tablet_bootstrap.cc:492] T 87f489b6b4e744c49e78570ba90508d5 P 8a284ede62614acfbb33e00a6291ed54: No bootstrap required, opened a new log
I20260926 09:31:14.612540 19443 ts_tablet_manager.cc:1397] T 87f489b6b4e744c49e78570ba90508d5 P 8a284ede62614acfbb33e00a6291ed54: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:31:14.614527 19446 tablet_bootstrap.cc:492] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd: Bootstrap starting.
I20260926 09:31:14.614979 19442 raft_consensus.cc:348] T 87f489b6b4e744c49e78570ba90508d5 P 9caf1d814b3b4e0a81b307d96a6e38c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f08afaa9ab4a00851c8d695a7dd9bd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45951 } } peers { permanent_uuid: "9caf1d814b3b4e0a81b307d96a6e38c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33177 } } peers { permanent_uuid: "8a284ede62614acfbb33e00a6291ed54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32925 } }
I20260926 09:31:14.615123 19446 tablet_bootstrap.cc:654] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:14.615155 19442 raft_consensus.cc:374] T 87f489b6b4e744c49e78570ba90508d5 P 9caf1d814b3b4e0a81b307d96a6e38c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:14.615190 19442 raft_consensus.cc:729] T 87f489b6b4e744c49e78570ba90508d5 P 9caf1d814b3b4e0a81b307d96a6e38c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9caf1d814b3b4e0a81b307d96a6e38c4, State: Initialized, Role: FOLLOWER
I20260926 09:31:14.615361 19442 consensus_queue.cc:260] T 87f489b6b4e744c49e78570ba90508d5 P 9caf1d814b3b4e0a81b307d96a6e38c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f08afaa9ab4a00851c8d695a7dd9bd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45951 } } peers { permanent_uuid: "9caf1d814b3b4e0a81b307d96a6e38c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33177 } } peers { permanent_uuid: "8a284ede62614acfbb33e00a6291ed54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32925 } }
I20260926 09:31:14.615360 19443 raft_consensus.cc:348] T 87f489b6b4e744c49e78570ba90508d5 P 8a284ede62614acfbb33e00a6291ed54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f08afaa9ab4a00851c8d695a7dd9bd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45951 } } peers { permanent_uuid: "9caf1d814b3b4e0a81b307d96a6e38c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33177 } } peers { permanent_uuid: "8a284ede62614acfbb33e00a6291ed54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32925 } }
I20260926 09:31:14.615468 19446 log.cc:824] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:14.615500 19443 raft_consensus.cc:374] T 87f489b6b4e744c49e78570ba90508d5 P 8a284ede62614acfbb33e00a6291ed54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:14.615540 19443 raft_consensus.cc:729] T 87f489b6b4e744c49e78570ba90508d5 P 8a284ede62614acfbb33e00a6291ed54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a284ede62614acfbb33e00a6291ed54, State: Initialized, Role: FOLLOWER
I20260926 09:31:14.615634 19442 ts_tablet_manager.cc:1428] T 87f489b6b4e744c49e78570ba90508d5 P 9caf1d814b3b4e0a81b307d96a6e38c4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:31:14.615741 19297 heartbeater.cc:503] Master 127.0.98.62:41293 was elected leader, sending a full tablet report...
I20260926 09:31:14.615718 19443 consensus_queue.cc:260] T 87f489b6b4e744c49e78570ba90508d5 P 8a284ede62614acfbb33e00a6291ed54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f08afaa9ab4a00851c8d695a7dd9bd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45951 } } peers { permanent_uuid: "9caf1d814b3b4e0a81b307d96a6e38c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33177 } } peers { permanent_uuid: "8a284ede62614acfbb33e00a6291ed54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32925 } }
I20260926 09:31:14.615970 19443 ts_tablet_manager.cc:1428] T 87f489b6b4e744c49e78570ba90508d5 P 8a284ede62614acfbb33e00a6291ed54: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:31:14.615991 19428 heartbeater.cc:503] Master 127.0.98.62:41293 was elected leader, sending a full tablet report...
I20260926 09:31:14.616240 19446 tablet_bootstrap.cc:492] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd: No bootstrap required, opened a new log
I20260926 09:31:14.616335 19446 ts_tablet_manager.cc:1397] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:31:14.618785 19446 raft_consensus.cc:348] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f08afaa9ab4a00851c8d695a7dd9bd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45951 } } peers { permanent_uuid: "9caf1d814b3b4e0a81b307d96a6e38c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33177 } } peers { permanent_uuid: "8a284ede62614acfbb33e00a6291ed54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32925 } }
I20260926 09:31:14.618937 19446 raft_consensus.cc:374] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:14.618988 19446 raft_consensus.cc:729] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f08afaa9ab4a00851c8d695a7dd9bd, State: Initialized, Role: FOLLOWER
I20260926 09:31:14.619102 19446 consensus_queue.cc:260] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f08afaa9ab4a00851c8d695a7dd9bd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45951 } } peers { permanent_uuid: "9caf1d814b3b4e0a81b307d96a6e38c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33177 } } peers { permanent_uuid: "8a284ede62614acfbb33e00a6291ed54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32925 } }
I20260926 09:31:14.619355 19446 ts_tablet_manager.cc:1428] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:31:14.619459 19166 heartbeater.cc:503] Master 127.0.98.62:41293 was elected leader, sending a full tablet report...
W20260926 09:31:14.699963 19298 tablet.cc:2369] T 87f489b6b4e744c49e78570ba90508d5 P 9caf1d814b3b4e0a81b307d96a6e38c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:31:14.734797 19450 raft_consensus.cc:482] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:31:14.734935 19450 raft_consensus.cc:504] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f08afaa9ab4a00851c8d695a7dd9bd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45951 } } peers { permanent_uuid: "9caf1d814b3b4e0a81b307d96a6e38c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33177 } } peers { permanent_uuid: "8a284ede62614acfbb33e00a6291ed54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32925 } }
I20260926 09:31:14.735409 19450 leader_election.cc:288] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9caf1d814b3b4e0a81b307d96a6e38c4 (127.0.98.2:33177), 8a284ede62614acfbb33e00a6291ed54 (127.0.98.3:32925)
I20260926 09:31:14.739590 19439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:14.735701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47407 (local address 127.0.98.3:32925)
0926 09:31:14.735759 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:14.735761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:14.735774 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:31:14.735815 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:14.735816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:14.735826 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:14.735928 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:14.736114 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.736954 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.738550 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.739132 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.739218 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:14.739229 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:14.739458 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 09:31:14.739516 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 09:31:14.739590 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:14.735697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46495 (local address 127.0.98.2:33177)
0926 09:31:14.735748 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:14.735750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:14.735902 (+ 152us) server_negotiation.cc:407] Connection header received
0926 09:31:14.735976 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:14.735977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:14.735987 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:14.736039 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:14.736220 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.736967 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.738551 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.739132 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.739218 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:14.739228 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:14.739458 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 09:31:14.739508 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 09:31:14.739564 19168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:14.735615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32925 (local address 127.0.98.1:47407)
0926 09:31:14.735691 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:14.735704 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:14.735749 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:14.735937 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:14.735937 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:14.735939 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:14.736013 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:14.736015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.737065 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.737066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:14.738429 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:14.738432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.739238 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.739239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:14.739337 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:14.739341 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:31:14.739408 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 09:31:14.739462 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":50}
I20260926 09:31:14.739571 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:14.735615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33177 (local address 127.0.98.1:46495)
0926 09:31:14.735834 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:14.735843 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:14.735907 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:14.736060 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:14.736060 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:14.736062 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:14.736127 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:14.736128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.737065 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.737066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:14.738432 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:14.738433 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.739238 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.739239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:14.739337 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:14.739342 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:31:14.739408 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 09:31:14.739460 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":95,"thread_start_us":102,"threads_started":1}
I20260926 09:31:14.740036 19382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f489b6b4e744c49e78570ba90508d5" candidate_uuid: "75f08afaa9ab4a00851c8d695a7dd9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a284ede62614acfbb33e00a6291ed54" is_pre_election: true
I20260926 09:31:14.740036 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f489b6b4e744c49e78570ba90508d5" candidate_uuid: "75f08afaa9ab4a00851c8d695a7dd9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9caf1d814b3b4e0a81b307d96a6e38c4" is_pre_election: true
I20260926 09:31:14.740207 19251 raft_consensus.cc:2454] T 87f489b6b4e744c49e78570ba90508d5 P 9caf1d814b3b4e0a81b307d96a6e38c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75f08afaa9ab4a00851c8d695a7dd9bd in term 0.
I20260926 09:31:14.740209 19382 raft_consensus.cc:2454] T 87f489b6b4e744c49e78570ba90508d5 P 8a284ede62614acfbb33e00a6291ed54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75f08afaa9ab4a00851c8d695a7dd9bd in term 0.
I20260926 09:31:14.740484 19054 leader_election.cc:302] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75f08afaa9ab4a00851c8d695a7dd9bd, 9caf1d814b3b4e0a81b307d96a6e38c4; no voters:
I20260926 09:31:14.740654 19450 raft_consensus.cc:2781] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:31:14.740763 19450 raft_consensus.cc:482] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:31:14.740779 19450 raft_consensus.cc:3037] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:14.741520 19450 raft_consensus.cc:504] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f08afaa9ab4a00851c8d695a7dd9bd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45951 } } peers { permanent_uuid: "9caf1d814b3b4e0a81b307d96a6e38c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33177 } } peers { permanent_uuid: "8a284ede62614acfbb33e00a6291ed54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32925 } }
I20260926 09:31:14.741703 19450 leader_election.cc:288] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd [CANDIDATE]: Term 1 election: Requested vote from peers 9caf1d814b3b4e0a81b307d96a6e38c4 (127.0.98.2:33177), 8a284ede62614acfbb33e00a6291ed54 (127.0.98.3:32925)
I20260926 09:31:14.741887 19382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f489b6b4e744c49e78570ba90508d5" candidate_uuid: "75f08afaa9ab4a00851c8d695a7dd9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a284ede62614acfbb33e00a6291ed54"
I20260926 09:31:14.741907 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f489b6b4e744c49e78570ba90508d5" candidate_uuid: "75f08afaa9ab4a00851c8d695a7dd9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9caf1d814b3b4e0a81b307d96a6e38c4"
I20260926 09:31:14.741972 19382 raft_consensus.cc:3037] T 87f489b6b4e744c49e78570ba90508d5 P 8a284ede62614acfbb33e00a6291ed54 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:14.742017 19251 raft_consensus.cc:3037] T 87f489b6b4e744c49e78570ba90508d5 P 9caf1d814b3b4e0a81b307d96a6e38c4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:14.742702 19382 raft_consensus.cc:2454] T 87f489b6b4e744c49e78570ba90508d5 P 8a284ede62614acfbb33e00a6291ed54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75f08afaa9ab4a00851c8d695a7dd9bd in term 1.
I20260926 09:31:14.742885 19055 leader_election.cc:302] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75f08afaa9ab4a00851c8d695a7dd9bd, 8a284ede62614acfbb33e00a6291ed54; no voters:
I20260926 09:31:14.742913 19251 raft_consensus.cc:2454] T 87f489b6b4e744c49e78570ba90508d5 P 9caf1d814b3b4e0a81b307d96a6e38c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75f08afaa9ab4a00851c8d695a7dd9bd in term 1.
I20260926 09:31:14.742993 19450 raft_consensus.cc:2781] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:14.743144 19450 raft_consensus.cc:686] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd [term 1 LEADER]: Becoming Leader. State: Replica: 75f08afaa9ab4a00851c8d695a7dd9bd, State: Running, Role: LEADER
I20260926 09:31:14.743341 19450 consensus_queue.cc:237] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f08afaa9ab4a00851c8d695a7dd9bd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45951 } } peers { permanent_uuid: "9caf1d814b3b4e0a81b307d96a6e38c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33177 } } peers { permanent_uuid: "8a284ede62614acfbb33e00a6291ed54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32925 } }
I20260926 09:31:14.744298 18977 catalog_manager.cc:5495] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 75f08afaa9ab4a00851c8d695a7dd9bd (127.0.98.1). New cstate: current_term: 1 leader_uuid: "75f08afaa9ab4a00851c8d695a7dd9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f08afaa9ab4a00851c8d695a7dd9bd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9caf1d814b3b4e0a81b307d96a6e38c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a284ede62614acfbb33e00a6291ed54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32925 } health_report { overall_health: UNKNOWN } } }
I20260926 09:31:14.752080 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:14.749419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50568 (local address 127.0.98.1:45951)
0926 09:31:14.749460 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:14.749461 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:14.749509 (+ 48us) server_negotiation.cc:407] Connection header received
0926 09:31:14.749607 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:14.749608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:14.749628 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:14.749706 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:14.749920 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.750604 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.751322 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:14.751510 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:14.751566 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:14.751696 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:14.751811 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:14.751886 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:14.751996 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 09:31:14.752006 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:31:14.753999 19382 raft_consensus.cc:1261] T 87f489b6b4e744c49e78570ba90508d5 P 8a284ede62614acfbb33e00a6291ed54 [term 1 FOLLOWER]: Refusing update from remote peer 75f08afaa9ab4a00851c8d695a7dd9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:31:14.753999 19251 raft_consensus.cc:1261] T 87f489b6b4e744c49e78570ba90508d5 P 9caf1d814b3b4e0a81b307d96a6e38c4 [term 1 FOLLOWER]: Refusing update from remote peer 75f08afaa9ab4a00851c8d695a7dd9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:31:14.754382 19452 consensus_queue.cc:1035] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a284ede62614acfbb33e00a6291ed54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:31:14.754551 19450 consensus_queue.cc:1035] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9caf1d814b3b4e0a81b307d96a6e38c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:31:14.757042 19455 mvcc.cc:205] Tried to move back new op lower bound from 7333540146189828096 to 7333540146150035456. Current Snapshot: MvccSnapshot[applied={T|T < 7333540146189828096}]
I20260926 09:31:14.757331 19456 mvcc.cc:205] Tried to move back new op lower bound from 7333540146189828096 to 7333540146150035456. Current Snapshot: MvccSnapshot[applied={T|T < 7333540146189828096}]
I20260926 09:31:14.757885 19457 mvcc.cc:205] Tried to move back new op lower bound from 7333540146189828096 to 7333540146150035456. Current Snapshot: MvccSnapshot[applied={T|T < 7333540146189828096 or (T in {7333540146189828096})}]
I20260926 09:31:14.759980 18977 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:31:14.760076 18977 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:31:14.760782 18977 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fd12df3a7bf42a49f037656c48d8bb4: Sending DeleteTablet for 3 replicas of tablet 87f489b6b4e744c49e78570ba90508d5
I20260926 09:31:14.761086 19100 tablet_service.cc:1514] Processing DeleteTablet for tablet 87f489b6b4e744c49e78570ba90508d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:31:14 UTC) from {username='slave'} at 127.0.0.1:50554
I20260926 09:31:14.761132 19362 tablet_service.cc:1514] Processing DeleteTablet for tablet 87f489b6b4e744c49e78570ba90508d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:31:14 UTC) from {username='slave'} at 127.0.0.1:39560
I20260926 09:31:14.761204 19231 tablet_service.cc:1514] Processing DeleteTablet for tablet 87f489b6b4e744c49e78570ba90508d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:31:14 UTC) from {username='slave'} at 127.0.0.1:45826
I20260926 09:31:14.761498 19465 tablet_replica.cc:330] T 87f489b6b4e744c49e78570ba90508d5 P 75f08afaa9ab4a00851c8d695a7dd9bd: stopping tablet replica
I20260926 09:31:14.761521 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 19038
I20260926 09:31:14.761348 19466 tablet_replica.cc:330] T 87f489b6b4e744c49e78570ba90508d5 P 8a284ede62614acfbb33e00a6291ed54: stopping tablet replica
I20260926 09:31:14.762276 19466 raft_consensus.cc:2229] T 87f489b6b4e744c49e78570ba90508d5 P 8a284ede62614acfbb33e00a6291ed54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:31:14.762382 19466 raft_consensus.cc:2258] T 87f489b6b4e744c49e78570ba90508d5 P 8a284ede62614acfbb33e00a6291ed54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:31:14.763114 19466 ts_tablet_manager.cc:1905] T 87f489b6b4e744c49e78570ba90508d5 P 8a284ede62614acfbb33e00a6291ed54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:31:14.764906 19466 ts_tablet_manager.cc:1918] T 87f489b6b4e744c49e78570ba90508d5 P 8a284ede62614acfbb33e00a6291ed54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:31:14.764971 19466 log.cc:1196] T 87f489b6b4e744c49e78570ba90508d5 P 8a284ede62614acfbb33e00a6291ed54: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/87f489b6b4e744c49e78570ba90508d5
I20260926 09:31:14.765244 19466 ts_tablet_manager.cc:1939] T 87f489b6b4e744c49e78570ba90508d5 P 8a284ede62614acfbb33e00a6291ed54: Deleting consensus metadata
I20260926 09:31:14.765722 18964 catalog_manager.cc:4841] TS 8a284ede62614acfbb33e00a6291ed54 (127.0.98.3:32925): tablet 87f489b6b4e744c49e78570ba90508d5 (table test-table [id=e68aa47995aa409c8fa8af152ec4bb40]) successfully deleted
W20260926 09:31:14.766640 18963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45951 (error 108)
I20260926 09:31:14.767130 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:14.766990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45951 (local address 127.0.0.1:50572)
0926 09:31:14.767050 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:14.767094 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 09:31:14.767130 19467 tablet_replica.cc:330] T 87f489b6b4e744c49e78570ba90508d5 P 9caf1d814b3b4e0a81b307d96a6e38c4: stopping tablet replica
W20260926 09:31:14.767242 18963 catalog_manager.cc:4584] TS 75f08afaa9ab4a00851c8d695a7dd9bd (127.0.98.1:45951): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 87f489b6b4e744c49e78570ba90508d5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45951: connect: Connection refused (error 111)
I20260926 09:31:14.767261 19467 raft_consensus.cc:2229] T 87f489b6b4e744c49e78570ba90508d5 P 9caf1d814b3b4e0a81b307d96a6e38c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:31:14.767405 19467 raft_consensus.cc:2258] T 87f489b6b4e744c49e78570ba90508d5 P 9caf1d814b3b4e0a81b307d96a6e38c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:31:14.767616 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 19169
I20260926 09:31:14.767843 19467 ts_tablet_manager.cc:1905] T 87f489b6b4e744c49e78570ba90508d5 P 9caf1d814b3b4e0a81b307d96a6e38c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:31:14.773113 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:14.772975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33177 (local address 127.0.0.1:45842)
0926 09:31:14.773042 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:14.773076 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 09:31:14.773097 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 19300
I20260926 09:31:14.777631 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 18947
2026-09-26T09:31:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (778 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:31:14 dist-test-slave-dfzm krb5kdc[19471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:31:14 dist-test-slave-dfzm krb5kdc[19471](info): set up 2 sockets
Sep 26 09:31:14 dist-test-slave-dfzm krb5kdc[19471](info): commencing operation
krb5kdc: starting...
W20260926 09:31:15.813539 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:31:15 dist-test-slave-dfzm krb5kdc[19471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:31:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:31:15.932135 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35135
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:16.014070 19484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:16.014283 19484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:16.014307 19484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:16.015448 19484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:16.015489 19484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:16.015504 19484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:16.015516 19484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:16.015528 19484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:16.017071 19484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:16.017316 19484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:16.017529 19484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:16.022418 19492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:16.022405 19489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:16.022440 19484 server_base.cc:1034] running on GCE node
W20260926 09:31:16.022406 19490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:16.022897 19484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:16.023185 19484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:16.024353 19484 hybrid_clock.cc:648] HybridClock initialized: now 1790415076024338 us; error 29 us; skew 500 ppm
Sep 26 09:31:16 dist-test-slave-dfzm krb5kdc[19471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:16.027143 19484 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:31:16.027640 19484 webserver.cc:460] Webserver started at http://127.0.98.62:36281/ using document root <none> and password file <none>
I20260926 09:31:16.027875 19484 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:16.027943 19484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:16.028095 19484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:16.029182 19484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "a3a92a4813f14460a5a4d1584a202715"
format_stamp: "Formatted at 2026-09-26 09:31:16 on dist-test-slave-dfzm"
server_key: "0a34d7c81b177dfda864ffe207149eaf"
server_key_iv: "9ed5e6ec8b59ed7cef5655f4b6e28574"
server_key_version: "encryptionkey@0"
I20260926 09:31:16.029563 19484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "a3a92a4813f14460a5a4d1584a202715"
format_stamp: "Formatted at 2026-09-26 09:31:16 on dist-test-slave-dfzm"
server_key: "0a34d7c81b177dfda864ffe207149eaf"
server_key_iv: "9ed5e6ec8b59ed7cef5655f4b6e28574"
server_key_version: "encryptionkey@0"
I20260926 09:31:16.030947 19484 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:16.031734 19499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:16.031919 19484 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:16.032018 19484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "a3a92a4813f14460a5a4d1584a202715"
format_stamp: "Formatted at 2026-09-26 09:31:16 on dist-test-slave-dfzm"
server_key: "0a34d7c81b177dfda864ffe207149eaf"
server_key_iv: "9ed5e6ec8b59ed7cef5655f4b6e28574"
server_key_version: "encryptionkey@0"
I20260926 09:31:16.032105 19484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:16.053575 19484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:16.055708 19484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:16.055877 19484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:16.060254 19484 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35135
I20260926 09:31:16.060303 19551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35135 every 8 connection(s)
I20260926 09:31:16.060685 19484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:16.061652 19552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:16.064007 19552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715: Bootstrap starting.
I20260926 09:31:16.064550 19552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:16.064803 19552 log.cc:824] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:16.065490 19552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715: No bootstrap required, opened a new log
I20260926 09:31:16.067054 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 19484
I20260926 09:31:16.067158 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:16.067170 19552 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a92a4813f14460a5a4d1584a202715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35135 } }
I20260926 09:31:16.067376 19552 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:16.067409 392 external_mini_cluster.cc:1428] Setting key 201efde2313d57d7824ed5c82d3eb485
I20260926 09:31:16.067426 19552 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3a92a4813f14460a5a4d1584a202715, State: Initialized, Role: FOLLOWER
I20260926 09:31:16.067597 19552 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a92a4813f14460a5a4d1584a202715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35135 } }
I20260926 09:31:16.067689 19552 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:31:16.067725 19552 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:31:16.068063 19552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:16.069180 19552 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a92a4813f14460a5a4d1584a202715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35135 } }
I20260926 09:31:16.069285 19552 leader_election.cc:302] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3a92a4813f14460a5a4d1584a202715; no voters:
I20260926 09:31:16.069453 19552 leader_election.cc:288] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:31:16.069520 19557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:16.069694 19557 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715 [term 1 LEADER]: Becoming Leader. State: Replica: a3a92a4813f14460a5a4d1584a202715, State: Running, Role: LEADER
I20260926 09:31:16.069720 19552 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:16.069844 19557 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a92a4813f14460a5a4d1584a202715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35135 } }
I20260926 09:31:16.071008 19559 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3a92a4813f14460a5a4d1584a202715. Latest consensus state: current_term: 1 leader_uuid: "a3a92a4813f14460a5a4d1584a202715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a92a4813f14460a5a4d1584a202715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35135 } } }
I20260926 09:31:16.071097 19559 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:16.071305 19558 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3a92a4813f14460a5a4d1584a202715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a92a4813f14460a5a4d1584a202715" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35135 } } }
I20260926 09:31:16.071592 19558 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715 [sys.catalog]: This master's current role is: LEADER
W20260926 09:31:16.071781 19572 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:31:16.071847 19572 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:31:16.071904 19573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:16.072072 19573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:16.073807 19573 catalog_manager.cc:1347] Generated new cluster ID: c8c46c7a243d43afb72006d0c3c76182
I20260926 09:31:16.073865 19573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:31:16 dist-test-slave-dfzm krb5kdc[19471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:31:16.078761 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:16.067901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56766 (local address 127.0.98.62:35135)
0926 09:31:16.068061 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:16.068071 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:16.068106 (+ 35us) server_negotiation.cc:407] Connection header received
0926 09:31:16.068220 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:16.068227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:16.068291 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:16.068444 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:16.069031 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.070365 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.071529 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.072135 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.072274 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:16.075340 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:16.075344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:16.075344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:16.075357 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:16.077143 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:16.077497 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:16.077500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:16.077501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:16.077632 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:16.077845 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:16.077846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:16.077846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:16.078133 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:16.078275 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:16.078421 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 09:31:16.078561 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260926 09:31:16.094950 19573 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:16.095691 19573 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:16.098134 19573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715: Generated new TSK 0
I20260926 09:31:16.098335 19573 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 09:31:16.128134 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35135
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:16.209048 19578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:16.209328 19578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:16.209369 19578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:16.210539 19578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:16.210613 19578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:16.210659 19578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:31:16.212164 19578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:16.212421 19578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:16.212648 19578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:31:16.217761 19578 server_base.cc:1034] running on GCE node
W20260926 09:31:16.217835 19583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:16.217876 19586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:16.217833 19584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:16.218242 19578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:16.218500 19578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:16.219682 19578 hybrid_clock.cc:648] HybridClock initialized: now 1790415076219661 us; error 35 us; skew 500 ppm
Sep 26 09:31:16 dist-test-slave-dfzm krb5kdc[19471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:16.222342 19578 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 09:31:16.222752 19578 webserver.cc:460] Webserver started at http://127.0.98.1:35763/ using document root <none> and password file <none>
I20260926 09:31:16.222967 19578 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:16.223078 19578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:16.223220 19578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:16.224253 19578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "d4b1edd394b74b7f8f64eab84427fc8f"
format_stamp: "Formatted at 2026-09-26 09:31:16 on dist-test-slave-dfzm"
server_key: "49a2fa922802a84273a6d1c8ceaaf595"
server_key_iv: "36a5d968f5975f67e11622ff999ee24f"
server_key_version: "encryptionkey@0"
I20260926 09:31:16.224612 19578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d4b1edd394b74b7f8f64eab84427fc8f"
format_stamp: "Formatted at 2026-09-26 09:31:16 on dist-test-slave-dfzm"
server_key: "49a2fa922802a84273a6d1c8ceaaf595"
server_key_iv: "36a5d968f5975f67e11622ff999ee24f"
server_key_version: "encryptionkey@0"
I20260926 09:31:16.226346 19578 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 09:31:16.227210 19593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:16.227427 19578 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:16.227650 19578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "d4b1edd394b74b7f8f64eab84427fc8f"
format_stamp: "Formatted at 2026-09-26 09:31:16 on dist-test-slave-dfzm"
server_key: "49a2fa922802a84273a6d1c8ceaaf595"
server_key_iv: "36a5d968f5975f67e11622ff999ee24f"
server_key_version: "encryptionkey@0"
I20260926 09:31:16.227737 19578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:16.249744 19578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:16.251767 19578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:16.251945 19578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:16.252246 19578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:31:16.252568 19578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:31:16.252624 19578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:16.252660 19578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:31:16.252686 19578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:16.258635 19578 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35203
I20260926 09:31:16.258689 19706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35203 every 8 connection(s)
I20260926 09:31:16.259019 19578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:31:16.263355 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 19578
I20260926 09:31:16.263505 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:31:16.263708 392 external_mini_cluster.cc:1428] Setting key 6388d0b802288268598cfbe2e480dfbf
Sep 26 09:31:16 dist-test-slave-dfzm krb5kdc[19471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:31:16.269865 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:16.259783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45291 (local address 127.0.98.62:35135)
0926 09:31:16.259828 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:16.259830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:16.260096 (+ 266us) server_negotiation.cc:407] Connection header received
0926 09:31:16.261064 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:16.261065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:16.261082 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:16.261136 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:16.261860 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.262382 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.263487 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.263754 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.263837 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:16.266413 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:16.266415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:16.266416 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:16.266425 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:16.268214 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:16.268647 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:16.268648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:16.268649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:16.268717 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:16.268979 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:16.268981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:16.268981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:16.269147 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:16.269229 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:16.269532 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 09:31:16.269671 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:31:16.269973 19709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:16.259781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35135 (local address 127.0.98.1:45291)
0926 09:31:16.259966 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:16.260002 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:16.260940 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:16.261211 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:16.261215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:16.261491 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:16.261766 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:16.261770 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.262482 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.262483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:16.263326 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:16.263328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.263909 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.263910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:16.264023 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:16.264604 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:16.264613 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:16.266317 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:16.268310 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:16.268316 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:16.268318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:16.268548 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:16.268795 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:16.268796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:16.268796 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:16.268906 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:16.269224 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:16.269229 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:16.269388 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 09:31:16.269476 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 09:31:16.269688 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260926 09:31:16.270500 19707 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35135
I20260926 09:31:16.270629 19707 heartbeater.cc:464] Registering TS with master...
I20260926 09:31:16.270812 19707 heartbeater.cc:511] Master 127.0.98.62:35135 requested a full tablet report, sending...
I20260926 09:31:16.271368 19516 ts_manager.cc:194] Registered new tserver with Master: d4b1edd394b74b7f8f64eab84427fc8f (127.0.98.1:35203)
I20260926 09:31:16.272101 19516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45291
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 09:31:16.292070 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35135
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:16.374069 19712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:16.374320 19712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:16.374361 19712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:16.375600 19712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:16.375679 19712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:16.375725 19712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:31:16.378651 19712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:16.378979 19712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:16.379266 19712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:16.385861 19720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:16.385891 19718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:16.385861 19717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:16.386348 19712 server_base.cc:1034] running on GCE node
I20260926 09:31:16.386533 19712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:16.386741 19712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:16.387889 19712 hybrid_clock.cc:648] HybridClock initialized: now 1790415076387869 us; error 35 us; skew 500 ppm
Sep 26 09:31:16 dist-test-slave-dfzm krb5kdc[19471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:16.390828 19712 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 09:31:16.391355 19712 webserver.cc:460] Webserver started at http://127.0.98.2:45501/ using document root <none> and password file <none>
I20260926 09:31:16.391568 19712 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:16.391616 19712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:16.391710 19712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:16.392822 19712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "33c5cb73c31446a8b2f5471b86db40fb"
format_stamp: "Formatted at 2026-09-26 09:31:16 on dist-test-slave-dfzm"
server_key: "45c5f06198e196a6c4dc54111431c957"
server_key_iv: "32f9328187524b9c6aabcc16dbe3612c"
server_key_version: "encryptionkey@0"
I20260926 09:31:16.393180 19712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "33c5cb73c31446a8b2f5471b86db40fb"
format_stamp: "Formatted at 2026-09-26 09:31:16 on dist-test-slave-dfzm"
server_key: "45c5f06198e196a6c4dc54111431c957"
server_key_iv: "32f9328187524b9c6aabcc16dbe3612c"
server_key_version: "encryptionkey@0"
I20260926 09:31:16.394718 19712 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:31:16.395591 19727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:16.395816 19712 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:31:16.395957 19712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "33c5cb73c31446a8b2f5471b86db40fb"
format_stamp: "Formatted at 2026-09-26 09:31:16 on dist-test-slave-dfzm"
server_key: "45c5f06198e196a6c4dc54111431c957"
server_key_iv: "32f9328187524b9c6aabcc16dbe3612c"
server_key_version: "encryptionkey@0"
I20260926 09:31:16.396063 19712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:16.406764 19712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:16.409358 19712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:16.409554 19712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:16.409920 19712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:31:16.410269 19712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:31:16.410307 19712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:16.410353 19712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:31:16.410388 19712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:16.416622 19712 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33353
I20260926 09:31:16.416702 19840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33353 every 8 connection(s)
I20260926 09:31:16.417037 19712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:31:16 dist-test-slave-dfzm krb5kdc[19471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:31:16.427134 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:16.417960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49367 (local address 127.0.98.62:35135)
0926 09:31:16.418029 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:16.418031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:16.418134 (+ 103us) server_negotiation.cc:407] Connection header received
0926 09:31:16.419182 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:16.419184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:16.419194 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:16.419258 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:16.419880 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.420381 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.421464 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.421736 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.421802 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:16.424290 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:16.424291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:16.424291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:16.424304 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:16.425630 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:16.426067 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:16.426068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:16.426068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:16.426133 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:16.426441 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:16.426442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:16.426442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:16.426566 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:16.426644 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:16.426973 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 09:31:16.427068 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 09:31:16.427285 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 19712
I20260926 09:31:16.427392 19843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:16.417838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35135 (local address 127.0.98.2:49367)
0926 09:31:16.418010 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:16.418052 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:16.419059 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:16.419324 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:16.419327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:16.419591 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:16.419787 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:16.419791 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.420505 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.420506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:16.421335 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:16.421337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.421818 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.421819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:16.421928 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:16.422555 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:16.422563 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:16.424200 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:16.425699 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:16.425705 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:16.425706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:16.425949 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:16.426225 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:16.426225 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:16.426226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:16.426356 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:16.426690 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:16.426696 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:16.426818 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 09:31:16.426917 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 09:31:16.427112 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260926 09:31:16.427402 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:31:16.427608 392 external_mini_cluster.cc:1428] Setting key 6fefda4bb2cbbc8ceef67e3b3e1be37d
I20260926 09:31:16.428390 19841 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35135
I20260926 09:31:16.428506 19841 heartbeater.cc:464] Registering TS with master...
I20260926 09:31:16.428704 19841 heartbeater.cc:511] Master 127.0.98.62:35135 requested a full tablet report, sending...
I20260926 09:31:16.429147 19515 ts_manager.cc:194] Registered new tserver with Master: 33c5cb73c31446a8b2f5471b86db40fb (127.0.98.2:33353)
I20260926 09:31:16.429543 19515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49367
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 09:31:16.453102 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35135
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:16.533653 19846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:16.533927 19846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:16.533994 19846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:16.535226 19846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:16.535292 19846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:16.535339 19846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:31:16.536885 19846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:16.537149 19846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:16.537379 19846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:16.542299 19854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:16.542347 19852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:16.542526 19846 server_base.cc:1034] running on GCE node
W20260926 09:31:16.542543 19851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:16.542797 19846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:16.543090 19846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:16.544294 19846 hybrid_clock.cc:648] HybridClock initialized: now 1790415076544255 us; error 56 us; skew 500 ppm
Sep 26 09:31:16 dist-test-slave-dfzm krb5kdc[19471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:16.547092 19846 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 09:31:16.547608 19846 webserver.cc:460] Webserver started at http://127.0.98.3:33667/ using document root <none> and password file <none>
I20260926 09:31:16.547837 19846 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:16.547904 19846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:16.548064 19846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:16.549117 19846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "14142eb8d53d4675877c876b07d2951c"
format_stamp: "Formatted at 2026-09-26 09:31:16 on dist-test-slave-dfzm"
server_key: "3fc5d9b6da7ff1203992d46172c2754b"
server_key_iv: "69248172094f9c74d486137bbaa1f2cb"
server_key_version: "encryptionkey@0"
I20260926 09:31:16.549475 19846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "14142eb8d53d4675877c876b07d2951c"
format_stamp: "Formatted at 2026-09-26 09:31:16 on dist-test-slave-dfzm"
server_key: "3fc5d9b6da7ff1203992d46172c2754b"
server_key_iv: "69248172094f9c74d486137bbaa1f2cb"
server_key_version: "encryptionkey@0"
I20260926 09:31:16.550964 19846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:31:16.551713 19861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:16.551882 19846 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:31:16.552018 19846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "14142eb8d53d4675877c876b07d2951c"
format_stamp: "Formatted at 2026-09-26 09:31:16 on dist-test-slave-dfzm"
server_key: "3fc5d9b6da7ff1203992d46172c2754b"
server_key_iv: "69248172094f9c74d486137bbaa1f2cb"
server_key_version: "encryptionkey@0"
I20260926 09:31:16.552105 19846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:16.565840 19846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:16.567884 19846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:16.568055 19846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:16.568290 19846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:31:16.568596 19846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:31:16.568651 19846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:16.568692 19846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:31:16.568722 19846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:16.575016 19846 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32929
I20260926 09:31:16.575099 19974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32929 every 8 connection(s)
I20260926 09:31:16.575415 19846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:31:16.578092 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 19846
I20260926 09:31:16.578215 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:31:16.578399 392 external_mini_cluster.cc:1428] Setting key 15eff39cf055db0a13b8fe4b58e85f61
Sep 26 09:31:16 dist-test-slave-dfzm krb5kdc[19471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:31:16.584656 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:16.576199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52339 (local address 127.0.98.62:35135)
0926 09:31:16.576264 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:16.576267 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:16.576455 (+ 188us) server_negotiation.cc:407] Connection header received
0926 09:31:16.577391 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:16.577392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:16.577402 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:16.577456 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:16.578120 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.578877 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.579925 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.580108 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.580164 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:16.582250 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:16.582251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:16.582252 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:16.582258 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:16.583437 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:16.583820 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:16.583821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:16.583821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:16.583869 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:16.584134 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:16.584135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:16.584135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:16.584233 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:16.584282 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:16.584525 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 09:31:16.584592 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 09:31:16.584853 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:16.576160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35135 (local address 127.0.98.3:52339)
0926 09:31:16.576327 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:16.576363 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:16.577287 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:16.577509 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:16.577512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:16.577833 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:16.578026 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:16.578031 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.579026 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.579027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:16.579850 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:16.579852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.580188 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.580189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:16.580288 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:16.580743 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:16.580750 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:16.582167 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:16.583518 (+ 1351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:16.583524 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:16.583525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:16.583753 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:16.583965 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:16.583966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:16.583966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:16.584074 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:16.584290 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:16.584294 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:16.584417 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 09:31:16.584501 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 09:31:16.584632 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260926 09:31:16.585310 19975 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35135
I20260926 09:31:16.585423 19975 heartbeater.cc:464] Registering TS with master...
I20260926 09:31:16.585640 19975 heartbeater.cc:511] Master 127.0.98.62:35135 requested a full tablet report, sending...
I20260926 09:31:16.586035 19515 ts_manager.cc:194] Registered new tserver with Master: 14142eb8d53d4675877c876b07d2951c (127.0.98.3:32929)
I20260926 09:31:16.586417 19515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52339
I20260926 09:31:16.590211 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:31:16.597955 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:16.591468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56778 (local address 127.0.98.62:35135)
0926 09:31:16.591549 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:16.591551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:16.591661 (+ 110us) server_negotiation.cc:407] Connection header received
0926 09:31:16.591738 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:16.591738 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:16.591749 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:16.591797 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:16.592341 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.592806 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.593612 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.593813 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.593916 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:16.595385 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:16.595387 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:16.595387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:16.595411 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:16.596735 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:16.597081 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:16.597082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:16.597083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:16.597144 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:16.597343 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:16.597344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:16.597344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:16.597505 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:16.597641 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:16.597772 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 09:31:16.597848 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 09:31:16.599704 19515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:31:16.601334 19515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:31:16.608047 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:16.603807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41160 (local address 127.0.98.1:35203)
0926 09:31:16.604011 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:16.604016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:16.604127 (+ 111us) server_negotiation.cc:407] Connection header received
0926 09:31:16.604265 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:16.604267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:16.604281 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:16.604363 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:16.604619 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.605671 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.606905 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.607613 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.607691 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:16.607749 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:16.607916 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 09:31:16.607961 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":84,"threads_started":1}
I20260926 09:31:16.608125 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:16.603681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33353 (local address 127.0.0.1:57878)
0926 09:31:16.603832 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:16.603873 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:16.603940 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:16.604185 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:16.604189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:16.604192 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:16.604259 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:16.604262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.605569 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.605570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:16.607010 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:16.607013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.607871 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.607872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:16.607934 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:16.607939 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:31:16.607997 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 09:31:16.608056 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":48,"threads_started":1}
I20260926 09:31:16.608309 19987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:16.603692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35203 (local address 127.0.0.1:41160)
0926 09:31:16.604062 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:16.604071 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:16.604155 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:16.604470 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:16.604471 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:16.604473 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:16.604518 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:16.604519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.605773 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.605774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:16.606806 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:16.606808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.607705 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.607706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:16.607785 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:16.607800 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 09:31:16.608185 (+ 385us) client_negotiation.cc:240] Negotiation successful
0926 09:31:16.608241 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":94,"threads_started":1}
I20260926 09:31:16.608340 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:16.603782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57878 (local address 127.0.98.2:33353)
0926 09:31:16.604015 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:16.604019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:16.604029 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:31:16.604070 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:16.604073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:16.604091 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:16.604173 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:16.604337 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.605467 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.607096 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.607773 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.607871 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:16.607928 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:16.608011 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 09:31:16.608053 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":125,"threads_started":1}
I20260926 09:31:16.609042 19641 tablet_service.cc:1467] Processing CreateTablet for tablet 1646d25793f548ec95859ed744d8d927 (DEFAULT_TABLE table=test-table [id=a4cf94e24a304f5099aaa2fc14ea9e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:31:16.609018 19775 tablet_service.cc:1467] Processing CreateTablet for tablet 1646d25793f548ec95859ed744d8d927 (DEFAULT_TABLE table=test-table [id=a4cf94e24a304f5099aaa2fc14ea9e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:31:16.609231 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:16.603886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32929 (local address 127.0.0.1:37340)
0926 09:31:16.604069 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:16.604252 (+ 183us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:16.604303 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:16.605896 (+ 1593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:16.605897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:16.605899 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:16.605986 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:16.605987 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.607173 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.607174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:16.608013 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:16.608015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.609033 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.609034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:16.609094 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:16.609098 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:31:16.609129 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 09:31:16.609168 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":43,"thread_start_us":69,"threads_started":1}
I20260926 09:31:16.609388 19641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1646d25793f548ec95859ed744d8d927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:16.609400 19775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1646d25793f548ec95859ed744d8d927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:16.609531 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:16.604112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37340 (local address 127.0.98.3:32929)
0926 09:31:16.605727 (+ 1615us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:16.605732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:16.605743 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:31:16.605780 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:16.605782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:16.605798 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:16.605885 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:16.606060 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.607087 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.608245 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.608929 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.609324 (+ 395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:16.609389 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:16.609417 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 09:31:16.609458 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1576,"thread_start_us":88,"threads_started":1}
I20260926 09:31:16.611865 19993 tablet_bootstrap.cc:492] T 1646d25793f548ec95859ed744d8d927 P d4b1edd394b74b7f8f64eab84427fc8f: Bootstrap starting.
I20260926 09:31:16.612073 19992 tablet_bootstrap.cc:492] T 1646d25793f548ec95859ed744d8d927 P 33c5cb73c31446a8b2f5471b86db40fb: Bootstrap starting.
I20260926 09:31:16.612717 19992 tablet_bootstrap.cc:654] T 1646d25793f548ec95859ed744d8d927 P 33c5cb73c31446a8b2f5471b86db40fb: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:16.612677 19897 tablet_service.cc:1467] Processing CreateTablet for tablet 1646d25793f548ec95859ed744d8d927 (DEFAULT_TABLE table=test-table [id=a4cf94e24a304f5099aaa2fc14ea9e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:31:16.612990 19897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1646d25793f548ec95859ed744d8d927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:16.613231 19992 log.cc:824] T 1646d25793f548ec95859ed744d8d927 P 33c5cb73c31446a8b2f5471b86db40fb: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:16.614027 19993 tablet_bootstrap.cc:654] T 1646d25793f548ec95859ed744d8d927 P d4b1edd394b74b7f8f64eab84427fc8f: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:16.614048 19992 tablet_bootstrap.cc:492] T 1646d25793f548ec95859ed744d8d927 P 33c5cb73c31446a8b2f5471b86db40fb: No bootstrap required, opened a new log
I20260926 09:31:16.614110 19992 ts_tablet_manager.cc:1397] T 1646d25793f548ec95859ed744d8d927 P 33c5cb73c31446a8b2f5471b86db40fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:31:16.614578 19993 log.cc:824] T 1646d25793f548ec95859ed744d8d927 P d4b1edd394b74b7f8f64eab84427fc8f: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:16.615311 19993 tablet_bootstrap.cc:492] T 1646d25793f548ec95859ed744d8d927 P d4b1edd394b74b7f8f64eab84427fc8f: No bootstrap required, opened a new log
I20260926 09:31:16.615378 19993 ts_tablet_manager.cc:1397] T 1646d25793f548ec95859ed744d8d927 P d4b1edd394b74b7f8f64eab84427fc8f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 09:31:16.616235 19996 tablet_bootstrap.cc:492] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c: Bootstrap starting.
I20260926 09:31:16.616169 19992 raft_consensus.cc:348] T 1646d25793f548ec95859ed744d8d927 P 33c5cb73c31446a8b2f5471b86db40fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c5cb73c31446a8b2f5471b86db40fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "d4b1edd394b74b7f8f64eab84427fc8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35203 } } peers { permanent_uuid: "14142eb8d53d4675877c876b07d2951c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32929 } }
I20260926 09:31:16.616361 19992 raft_consensus.cc:374] T 1646d25793f548ec95859ed744d8d927 P 33c5cb73c31446a8b2f5471b86db40fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:16.616415 19992 raft_consensus.cc:729] T 1646d25793f548ec95859ed744d8d927 P 33c5cb73c31446a8b2f5471b86db40fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c5cb73c31446a8b2f5471b86db40fb, State: Initialized, Role: FOLLOWER
I20260926 09:31:16.616550 19992 consensus_queue.cc:260] T 1646d25793f548ec95859ed744d8d927 P 33c5cb73c31446a8b2f5471b86db40fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c5cb73c31446a8b2f5471b86db40fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "d4b1edd394b74b7f8f64eab84427fc8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35203 } } peers { permanent_uuid: "14142eb8d53d4675877c876b07d2951c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32929 } }
I20260926 09:31:16.616843 19992 ts_tablet_manager.cc:1428] T 1646d25793f548ec95859ed744d8d927 P 33c5cb73c31446a8b2f5471b86db40fb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:31:16.616840 19841 heartbeater.cc:503] Master 127.0.98.62:35135 was elected leader, sending a full tablet report...
I20260926 09:31:16.617075 19996 tablet_bootstrap.cc:654] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:16.617146 19993 raft_consensus.cc:348] T 1646d25793f548ec95859ed744d8d927 P d4b1edd394b74b7f8f64eab84427fc8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c5cb73c31446a8b2f5471b86db40fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "d4b1edd394b74b7f8f64eab84427fc8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35203 } } peers { permanent_uuid: "14142eb8d53d4675877c876b07d2951c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32929 } }
I20260926 09:31:16.617343 19993 raft_consensus.cc:374] T 1646d25793f548ec95859ed744d8d927 P d4b1edd394b74b7f8f64eab84427fc8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:16.617393 19993 raft_consensus.cc:729] T 1646d25793f548ec95859ed744d8d927 P d4b1edd394b74b7f8f64eab84427fc8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b1edd394b74b7f8f64eab84427fc8f, State: Initialized, Role: FOLLOWER
I20260926 09:31:16.617516 19993 consensus_queue.cc:260] T 1646d25793f548ec95859ed744d8d927 P d4b1edd394b74b7f8f64eab84427fc8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c5cb73c31446a8b2f5471b86db40fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "d4b1edd394b74b7f8f64eab84427fc8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35203 } } peers { permanent_uuid: "14142eb8d53d4675877c876b07d2951c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32929 } }
I20260926 09:31:16.617661 19996 log.cc:824] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:16.617826 19993 ts_tablet_manager.cc:1428] T 1646d25793f548ec95859ed744d8d927 P d4b1edd394b74b7f8f64eab84427fc8f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 09:31:16.617868 19707 heartbeater.cc:503] Master 127.0.98.62:35135 was elected leader, sending a full tablet report...
I20260926 09:31:16.618491 19996 tablet_bootstrap.cc:492] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c: No bootstrap required, opened a new log
I20260926 09:31:16.618558 19996 ts_tablet_manager.cc:1397] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:31:16.620079 19996 raft_consensus.cc:348] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c5cb73c31446a8b2f5471b86db40fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "d4b1edd394b74b7f8f64eab84427fc8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35203 } } peers { permanent_uuid: "14142eb8d53d4675877c876b07d2951c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32929 } }
I20260926 09:31:16.620203 19996 raft_consensus.cc:374] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:16.620224 19996 raft_consensus.cc:729] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14142eb8d53d4675877c876b07d2951c, State: Initialized, Role: FOLLOWER
I20260926 09:31:16.620335 19996 consensus_queue.cc:260] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c5cb73c31446a8b2f5471b86db40fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "d4b1edd394b74b7f8f64eab84427fc8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35203 } } peers { permanent_uuid: "14142eb8d53d4675877c876b07d2951c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32929 } }
I20260926 09:31:16.620549 19996 ts_tablet_manager.cc:1428] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:31:16.620615 19975 heartbeater.cc:503] Master 127.0.98.62:35135 was elected leader, sending a full tablet report...
W20260926 09:31:16.667702 19842 tablet.cc:2369] T 1646d25793f548ec95859ed744d8d927 P 33c5cb73c31446a8b2f5471b86db40fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:31:16.759917 19708 tablet.cc:2369] T 1646d25793f548ec95859ed744d8d927 P d4b1edd394b74b7f8f64eab84427fc8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:31:16.789249 20000 raft_consensus.cc:482] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:31:16.789373 20000 raft_consensus.cc:504] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c5cb73c31446a8b2f5471b86db40fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "d4b1edd394b74b7f8f64eab84427fc8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35203 } } peers { permanent_uuid: "14142eb8d53d4675877c876b07d2951c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32929 } }
I20260926 09:31:16.789785 20000 leader_election.cc:288] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33c5cb73c31446a8b2f5471b86db40fb (127.0.98.2:33353), d4b1edd394b74b7f8f64eab84427fc8f (127.0.98.1:35203)
I20260926 09:31:16.793344 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:16.789952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33353 (local address 127.0.98.3:33471)
0926 09:31:16.789998 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:16.790011 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:16.790059 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:16.790250 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:16.790251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:16.790252 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:16.790353 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:16.790355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.791159 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.791160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:16.792169 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:16.792172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.793030 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.793031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:16.793138 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:16.793142 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:31:16.793216 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 09:31:16.793261 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 09:31:16.793354 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:16.790039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33471 (local address 127.0.98.2:33353)
0926 09:31:16.790110 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:16.790112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:16.790122 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:31:16.790150 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:16.790151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:16.790165 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:16.790247 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:16.790494 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.791076 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.792277 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.792936 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.793018 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:16.793029 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:16.793234 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 09:31:16.793295 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 09:31:16.793478 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:16.789958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35203 (local address 127.0.98.3:52613)
0926 09:31:16.790213 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:16.790223 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:16.790282 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:16.790495 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:16.790496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:16.790499 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:16.790569 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:16.790570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.791311 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.791312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:16.792256 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:16.792259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.793181 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.793182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:16.793264 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:16.793268 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:31:16.793347 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 09:31:16.793390 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":67,"thread_start_us":105,"threads_started":1}
I20260926 09:31:16.793464 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:16.790114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52613 (local address 127.0.98.1:35203)
0926 09:31:16.790167 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:16.790169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:16.790292 (+ 123us) server_negotiation.cc:407] Connection header received
0926 09:31:16.790342 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:16.790343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:16.790353 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:16.790414 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:16.790679 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.791211 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.792378 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.793053 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.793132 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:16.793143 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:16.793360 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 09:31:16.793399 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:31:16.793800 19795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1646d25793f548ec95859ed744d8d927" candidate_uuid: "14142eb8d53d4675877c876b07d2951c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33c5cb73c31446a8b2f5471b86db40fb" is_pre_election: true
I20260926 09:31:16.793869 19661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1646d25793f548ec95859ed744d8d927" candidate_uuid: "14142eb8d53d4675877c876b07d2951c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4b1edd394b74b7f8f64eab84427fc8f" is_pre_election: true
I20260926 09:31:16.793970 19795 raft_consensus.cc:2454] T 1646d25793f548ec95859ed744d8d927 P 33c5cb73c31446a8b2f5471b86db40fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14142eb8d53d4675877c876b07d2951c in term 0.
I20260926 09:31:16.794045 19661 raft_consensus.cc:2454] T 1646d25793f548ec95859ed744d8d927 P d4b1edd394b74b7f8f64eab84427fc8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14142eb8d53d4675877c876b07d2951c in term 0.
I20260926 09:31:16.794186 19865 leader_election.cc:302] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14142eb8d53d4675877c876b07d2951c, 33c5cb73c31446a8b2f5471b86db40fb; no voters:
I20260926 09:31:16.794363 20000 raft_consensus.cc:2781] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:31:16.794421 20000 raft_consensus.cc:482] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:31:16.794469 20000 raft_consensus.cc:3037] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:16.795214 20000 raft_consensus.cc:504] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c5cb73c31446a8b2f5471b86db40fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "d4b1edd394b74b7f8f64eab84427fc8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35203 } } peers { permanent_uuid: "14142eb8d53d4675877c876b07d2951c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32929 } }
I20260926 09:31:16.795442 20000 leader_election.cc:288] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c [CANDIDATE]: Term 1 election: Requested vote from peers 33c5cb73c31446a8b2f5471b86db40fb (127.0.98.2:33353), d4b1edd394b74b7f8f64eab84427fc8f (127.0.98.1:35203)
I20260926 09:31:16.795542 19661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1646d25793f548ec95859ed744d8d927" candidate_uuid: "14142eb8d53d4675877c876b07d2951c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4b1edd394b74b7f8f64eab84427fc8f"
I20260926 09:31:16.795622 19661 raft_consensus.cc:3037] T 1646d25793f548ec95859ed744d8d927 P d4b1edd394b74b7f8f64eab84427fc8f [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:16.795619 19795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1646d25793f548ec95859ed744d8d927" candidate_uuid: "14142eb8d53d4675877c876b07d2951c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33c5cb73c31446a8b2f5471b86db40fb"
I20260926 09:31:16.795688 19795 raft_consensus.cc:3037] T 1646d25793f548ec95859ed744d8d927 P 33c5cb73c31446a8b2f5471b86db40fb [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:16.796339 19661 raft_consensus.cc:2454] T 1646d25793f548ec95859ed744d8d927 P d4b1edd394b74b7f8f64eab84427fc8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14142eb8d53d4675877c876b07d2951c in term 1.
I20260926 09:31:16.796456 19795 raft_consensus.cc:2454] T 1646d25793f548ec95859ed744d8d927 P 33c5cb73c31446a8b2f5471b86db40fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14142eb8d53d4675877c876b07d2951c in term 1.
I20260926 09:31:16.796559 19863 leader_election.cc:302] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14142eb8d53d4675877c876b07d2951c, d4b1edd394b74b7f8f64eab84427fc8f; no voters:
I20260926 09:31:16.796655 20000 raft_consensus.cc:2781] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:16.796852 20000 raft_consensus.cc:686] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c [term 1 LEADER]: Becoming Leader. State: Replica: 14142eb8d53d4675877c876b07d2951c, State: Running, Role: LEADER
I20260926 09:31:16.797066 20000 consensus_queue.cc:237] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c5cb73c31446a8b2f5471b86db40fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "d4b1edd394b74b7f8f64eab84427fc8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35203 } } peers { permanent_uuid: "14142eb8d53d4675877c876b07d2951c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32929 } }
I20260926 09:31:16.798167 19515 catalog_manager.cc:5495] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c reported cstate change: term changed from 0 to 1, leader changed from <none> to 14142eb8d53d4675877c876b07d2951c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "14142eb8d53d4675877c876b07d2951c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c5cb73c31446a8b2f5471b86db40fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4b1edd394b74b7f8f64eab84427fc8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14142eb8d53d4675877c876b07d2951c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32929 } health_report { overall_health: HEALTHY } } }
W20260926 09:31:16.825991 19976 tablet.cc:2369] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:31:16.837874 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:16.835487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.0.98.3:32929)
0926 09:31:16.835552 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:16.835555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:16.835578 (+ 23us) server_negotiation.cc:407] Connection header received
0926 09:31:16.835619 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:16.835620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:16.835646 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:16.835750 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:16.835897 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.836421 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.837131 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:16.837329 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:16.837385 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:16.837515 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:16.837635 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:16.837695 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:16.837725 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 09:31:16.837741 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 09:31:16.839599 19795 raft_consensus.cc:1261] T 1646d25793f548ec95859ed744d8d927 P 33c5cb73c31446a8b2f5471b86db40fb [term 1 FOLLOWER]: Refusing update from remote peer 14142eb8d53d4675877c876b07d2951c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:31:16.839600 19661 raft_consensus.cc:1261] T 1646d25793f548ec95859ed744d8d927 P d4b1edd394b74b7f8f64eab84427fc8f [term 1 FOLLOWER]: Refusing update from remote peer 14142eb8d53d4675877c876b07d2951c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:31:16.839987 20002 consensus_queue.cc:1035] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c [LEADER]: Connected to new peer: Peer: permanent_uuid: "33c5cb73c31446a8b2f5471b86db40fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:31:16.840123 20000 consensus_queue.cc:1035] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4b1edd394b74b7f8f64eab84427fc8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:31:16.842437 20005 mvcc.cc:205] Tried to move back new op lower bound from 7333540154732916736 to 7333540154562207744. Current Snapshot: MvccSnapshot[applied={T|T < 7333540154732916736}]
I20260926 09:31:16.842913 20006 mvcc.cc:205] Tried to move back new op lower bound from 7333540154732916736 to 7333540154562207744. Current Snapshot: MvccSnapshot[applied={T|T < 7333540154732916736}]
I20260926 09:31:16.843312 20007 mvcc.cc:205] Tried to move back new op lower bound from 7333540154732916736 to 7333540154562207744. Current Snapshot: MvccSnapshot[applied={T|T < 7333540154732916736}]
I20260926 09:31:16.845112 19516 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:31:16.845212 19516 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:31:16.846114 19516 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3a92a4813f14460a5a4d1584a202715: Sending DeleteTablet for 3 replicas of tablet 1646d25793f548ec95859ed744d8d927
I20260926 09:31:16.846465 19775 tablet_service.cc:1514] Processing DeleteTablet for tablet 1646d25793f548ec95859ed744d8d927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:31:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57878
I20260926 09:31:16.846532 19641 tablet_service.cc:1514] Processing DeleteTablet for tablet 1646d25793f548ec95859ed744d8d927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:31:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41160
I20260926 09:31:16.846563 19897 tablet_service.cc:1514] Processing DeleteTablet for tablet 1646d25793f548ec95859ed744d8d927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:31:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37340
I20260926 09:31:16.846803 20016 tablet_replica.cc:330] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c: stopping tablet replica
I20260926 09:31:16.846913 20016 raft_consensus.cc:2229] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:31:16.847014 20016 raft_consensus.cc:2258] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:31:16.847151 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 19578
I20260926 09:31:16.848205 20016 ts_tablet_manager.cc:1905] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:31:16.850441 20016 ts_tablet_manager.cc:1918] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:31:16.850525 20016 log.cc:1196] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/1646d25793f548ec95859ed744d8d927
I20260926 09:31:16.850666 20014 tablet_replica.cc:330] T 1646d25793f548ec95859ed744d8d927 P 33c5cb73c31446a8b2f5471b86db40fb: stopping tablet replica
I20260926 09:31:16.850805 20014 raft_consensus.cc:2229] T 1646d25793f548ec95859ed744d8d927 P 33c5cb73c31446a8b2f5471b86db40fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:31:16.850837 20016 ts_tablet_manager.cc:1939] T 1646d25793f548ec95859ed744d8d927 P 14142eb8d53d4675877c876b07d2951c: Deleting consensus metadata
I20260926 09:31:16.850939 20014 raft_consensus.cc:2258] T 1646d25793f548ec95859ed744d8d927 P 33c5cb73c31446a8b2f5471b86db40fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:31:16.851302 19501 catalog_manager.cc:4841] TS 14142eb8d53d4675877c876b07d2951c (127.0.98.3:32929): tablet 1646d25793f548ec95859ed744d8d927 (table test-table [id=a4cf94e24a304f5099aaa2fc14ea9e01]) successfully deleted
I20260926 09:31:16.851459 20014 ts_tablet_manager.cc:1905] T 1646d25793f548ec95859ed744d8d927 P 33c5cb73c31446a8b2f5471b86db40fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:31:16.853076 20014 ts_tablet_manager.cc:1918] T 1646d25793f548ec95859ed744d8d927 P 33c5cb73c31446a8b2f5471b86db40fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:31:16.853153 20014 log.cc:1196] T 1646d25793f548ec95859ed744d8d927 P 33c5cb73c31446a8b2f5471b86db40fb: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414839069479-392-0/minicluster-data/ts-1/wal/wals/1646d25793f548ec95859ed744d8d927
I20260926 09:31:16.853420 20014 ts_tablet_manager.cc:1939] T 1646d25793f548ec95859ed744d8d927 P 33c5cb73c31446a8b2f5471b86db40fb: Deleting consensus metadata
I20260926 09:31:16.854061 19503 catalog_manager.cc:4841] TS 33c5cb73c31446a8b2f5471b86db40fb (127.0.98.2:33353): tablet 1646d25793f548ec95859ed744d8d927 (table test-table [id=a4cf94e24a304f5099aaa2fc14ea9e01]) successfully deleted
W20260926 09:31:16.854827 19501 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35203 (error 108)
I20260926 09:31:16.855055 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 19712
I20260926 09:31:16.856958 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:16.856823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35203 (local address 127.0.0.1:41162)
0926 09:31:16.856872 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:16.856913 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 09:31:16.857299 19501 catalog_manager.cc:4584] TS d4b1edd394b74b7f8f64eab84427fc8f (127.0.98.1:35203): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1646d25793f548ec95859ed744d8d927: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35203: connect: Connection refused (error 111)
I20260926 09:31:16.860868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 19846
I20260926 09:31:16.865371 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 19484
2026-09-26T09:31:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2088 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T09:31:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:16Z Disabled control of system clock
I20260926 09:31:16.892598 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44025
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 09:31:16.973076 20021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:16.973387 20021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:16.973429 20021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:16.974694 20021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:16.974745 20021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:16.974758 20021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:16.974771 20021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:16.974782 20021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:16.976171 20021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44025
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:16.976435 20021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:16.976742 20021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:16.981629 20029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:16.981665 20027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:16.981654 20026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:16.981878 20021 server_base.cc:1034] running on GCE node
I20260926 09:31:16.982129 20021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:16.982446 20021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:16.983620 20021 hybrid_clock.cc:648] HybridClock initialized: now 1790415076983596 us; error 37 us; skew 500 ppm
I20260926 09:31:16.985000 20021 webserver.cc:460] Webserver started at http://127.0.98.62:39469/ using document root <none> and password file <none>
I20260926 09:31:16.985249 20021 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:16.985298 20021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:16.985456 20021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:16.986557 20021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "bb71d581c2504fdf8d9d80faabdec7c2"
format_stamp: "Formatted at 2026-09-26 09:31:16 on dist-test-slave-dfzm"
server_key: "5f385964d723ac6a7931c975cd3cdd5e"
server_key_iv: "8f008e8cf61c65d2bfea0d48a00d4acb"
server_key_version: "encryptionkey@0"
I20260926 09:31:16.986919 20021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "bb71d581c2504fdf8d9d80faabdec7c2"
format_stamp: "Formatted at 2026-09-26 09:31:16 on dist-test-slave-dfzm"
server_key: "5f385964d723ac6a7931c975cd3cdd5e"
server_key_iv: "8f008e8cf61c65d2bfea0d48a00d4acb"
server_key_version: "encryptionkey@0"
I20260926 09:31:16.988161 20021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:31:16.988881 20035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:16.989122 20021 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:16.989221 20021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "bb71d581c2504fdf8d9d80faabdec7c2"
format_stamp: "Formatted at 2026-09-26 09:31:16 on dist-test-slave-dfzm"
server_key: "5f385964d723ac6a7931c975cd3cdd5e"
server_key_iv: "8f008e8cf61c65d2bfea0d48a00d4acb"
server_key_version: "encryptionkey@0"
I20260926 09:31:16.989296 20021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:17.003700 20021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:17.004047 20021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:17.004207 20021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:17.008272 20021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44025
I20260926 09:31:17.008322 20087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44025 every 8 connection(s)
I20260926 09:31:17.008658 20021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:17.009653 20088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:17.011936 20088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb71d581c2504fdf8d9d80faabdec7c2: Bootstrap starting.
I20260926 09:31:17.012511 20088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb71d581c2504fdf8d9d80faabdec7c2: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:17.012811 20088 log.cc:824] T 00000000000000000000000000000000 P bb71d581c2504fdf8d9d80faabdec7c2: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:17.013471 20088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb71d581c2504fdf8d9d80faabdec7c2: No bootstrap required, opened a new log
I20260926 09:31:17.015060 20088 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb71d581c2504fdf8d9d80faabdec7c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb71d581c2504fdf8d9d80faabdec7c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44025 } }
I20260926 09:31:17.015196 20088 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb71d581c2504fdf8d9d80faabdec7c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:17.015218 20088 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb71d581c2504fdf8d9d80faabdec7c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb71d581c2504fdf8d9d80faabdec7c2, State: Initialized, Role: FOLLOWER
I20260926 09:31:17.015393 20088 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb71d581c2504fdf8d9d80faabdec7c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb71d581c2504fdf8d9d80faabdec7c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44025 } }
I20260926 09:31:17.015455 20088 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb71d581c2504fdf8d9d80faabdec7c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:31:17.015476 20088 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb71d581c2504fdf8d9d80faabdec7c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:31:17.015547 20088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb71d581c2504fdf8d9d80faabdec7c2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:17.016045 20088 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb71d581c2504fdf8d9d80faabdec7c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb71d581c2504fdf8d9d80faabdec7c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44025 } }
I20260926 09:31:17.016157 20088 leader_election.cc:302] T 00000000000000000000000000000000 P bb71d581c2504fdf8d9d80faabdec7c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb71d581c2504fdf8d9d80faabdec7c2; no voters:
I20260926 09:31:17.016348 20088 leader_election.cc:288] T 00000000000000000000000000000000 P bb71d581c2504fdf8d9d80faabdec7c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:31:17.016396 20091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb71d581c2504fdf8d9d80faabdec7c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:17.016573 20091 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb71d581c2504fdf8d9d80faabdec7c2 [term 1 LEADER]: Becoming Leader. State: Replica: bb71d581c2504fdf8d9d80faabdec7c2, State: Running, Role: LEADER
I20260926 09:31:17.016618 20088 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb71d581c2504fdf8d9d80faabdec7c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:17.016717 20091 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb71d581c2504fdf8d9d80faabdec7c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb71d581c2504fdf8d9d80faabdec7c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44025 } }
I20260926 09:31:17.017077 20093 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb71d581c2504fdf8d9d80faabdec7c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb71d581c2504fdf8d9d80faabdec7c2. Latest consensus state: current_term: 1 leader_uuid: "bb71d581c2504fdf8d9d80faabdec7c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb71d581c2504fdf8d9d80faabdec7c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44025 } } }
I20260926 09:31:17.017159 20092 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb71d581c2504fdf8d9d80faabdec7c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb71d581c2504fdf8d9d80faabdec7c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb71d581c2504fdf8d9d80faabdec7c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44025 } } }
I20260926 09:31:17.017184 20093 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb71d581c2504fdf8d9d80faabdec7c2 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:17.017208 20092 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb71d581c2504fdf8d9d80faabdec7c2 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:17.017400 20100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:17.017449 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 20021
I20260926 09:31:17.017571 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:17.017858 392 external_mini_cluster.cc:1428] Setting key 7512734efd098640531be35fe716f774
I20260926 09:31:17.018235 20100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:17.019884 20100 catalog_manager.cc:1347] Generated new cluster ID: 90214342510549399b3b96dcdf677343
I20260926 09:31:17.019937 20100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:31:17.024523 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:17.018548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53602 (local address 127.0.98.62:44025)
0926 09:31:17.018828 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:17.018836 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:17.018874 (+ 38us) server_negotiation.cc:407] Connection header received
0926 09:31:17.019355 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:17.019363 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:17.019439 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:17.019591 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:17.019881 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.021228 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.022384 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.022987 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.023096 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:17.023831 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:17.023834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:17.023835 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:17.024030 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:17.024069 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:17.024082 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:17.024145 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:17.024242 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 09:31:17.024315 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":88,"threads_started":1}
I20260926 09:31:17.029000 20100 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:17.029522 20100 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:17.042732 20100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb71d581c2504fdf8d9d80faabdec7c2: Generated new TSK 0
I20260926 09:31:17.042960 20100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:31:17.045657 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44025
--builtin_ntp_servers=127.0.98.20:42921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 09:31:17.127734 20112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:17.127946 20112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:17.127969 20112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:17.129120 20112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:17.129170 20112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:17.129195 20112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 09:31:17.130654 20112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:17.130899 20112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:17.131215 20112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:17.137236 20120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:17.137226 20118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:17.137609 20112 server_base.cc:1034] running on GCE node
W20260926 09:31:17.137226 20117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:17.137894 20112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:17.138180 20112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:17.139354 20112 hybrid_clock.cc:648] HybridClock initialized: now 1790415077139328 us; error 44 us; skew 500 ppm
I20260926 09:31:17.140772 20112 webserver.cc:460] Webserver started at http://127.0.98.1:43481/ using document root <none> and password file <none>
I20260926 09:31:17.141022 20112 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:17.141077 20112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:17.141204 20112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:17.142696 20112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-0/data/instance:
uuid: "f3165536400e4eab9bee7d92d885b677"
format_stamp: "Formatted at 2026-09-26 09:31:17 on dist-test-slave-dfzm"
server_key: "e8eb44be9b0e314d03daf7128873544e"
server_key_iv: "8b1da04eed3383af9e945299984ed697"
server_key_version: "encryptionkey@0"
I20260926 09:31:17.143100 20112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f3165536400e4eab9bee7d92d885b677"
format_stamp: "Formatted at 2026-09-26 09:31:17 on dist-test-slave-dfzm"
server_key: "e8eb44be9b0e314d03daf7128873544e"
server_key_iv: "8b1da04eed3383af9e945299984ed697"
server_key_version: "encryptionkey@0"
I20260926 09:31:17.144937 20112 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 09:31:17.145989 20126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:17.146236 20112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:17.146335 20112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-0/wal
uuid: "f3165536400e4eab9bee7d92d885b677"
format_stamp: "Formatted at 2026-09-26 09:31:17 on dist-test-slave-dfzm"
server_key: "e8eb44be9b0e314d03daf7128873544e"
server_key_iv: "8b1da04eed3383af9e945299984ed697"
server_key_version: "encryptionkey@0"
I20260926 09:31:17.146411 20112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:17.176179 20112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:17.176463 20112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:17.176568 20112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:17.176770 20112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:31:17.177119 20112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:31:17.177153 20112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:17.177209 20112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:31:17.177242 20112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:17.183064 20112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36033
I20260926 09:31:17.183143 20239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36033 every 8 connection(s)
I20260926 09:31:17.183465 20112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-0/data/info.pb
I20260926 09:31:17.188789 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:17.184216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51683 (local address 127.0.98.62:44025)
0926 09:31:17.184291 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:17.184293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:17.184518 (+ 225us) server_negotiation.cc:407] Connection header received
0926 09:31:17.185044 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:17.185046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:17.185061 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:17.185129 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:17.185889 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.186392 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.187396 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.187594 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.187680 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:17.188480 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:17.188481 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:17.188482 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:17.188489 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:17.188504 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:17.188509 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:17.188578 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:17.188673 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 09:31:17.188705 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 09:31:17.189021 20242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:17.184220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44025 (local address 127.0.98.1:51683)
0926 09:31:17.184398 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:17.184432 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:17.184937 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:17.185209 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:17.185212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:17.185220 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:17.185794 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:17.185800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.186497 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.186498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:17.187288 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:17.187290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.187688 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.187689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:17.187823 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:17.188341 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:31:17.188341 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:17.188369 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:31:17.188392 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:17.188577 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:17.188584 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:17.188587 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:31:17.188676 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 09:31:17.188777 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260926 09:31:17.189574 20240 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44025
I20260926 09:31:17.189735 20240 heartbeater.cc:464] Registering TS with master...
I20260926 09:31:17.189921 20240 heartbeater.cc:511] Master 127.0.98.62:44025 requested a full tablet report, sending...
I20260926 09:31:17.190428 20052 ts_manager.cc:194] Registered new tserver with Master: f3165536400e4eab9bee7d92d885b677 (127.0.98.1:36033)
I20260926 09:31:17.190783 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 20112
I20260926 09:31:17.190896 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-0/wal/instance
I20260926 09:31:17.191092 392 external_mini_cluster.cc:1428] Setting key c2c16e94b1241b6729f0dd38a2597e64
I20260926 09:31:17.191207 20052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:51683
I20260926 09:31:17.192188 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44025
--builtin_ntp_servers=127.0.98.20:42921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 09:31:17.275130 20243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:17.275336 20243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:17.275358 20243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:17.276445 20243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:17.276487 20243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:17.276513 20243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 09:31:17.277869 20243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:17.278062 20243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:17.278308 20243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:17.283079 20249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:17.283095 20248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:17.283324 20243 server_base.cc:1034] running on GCE node
W20260926 09:31:17.283465 20251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:17.283717 20243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:17.283987 20243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:17.285149 20243 hybrid_clock.cc:648] HybridClock initialized: now 1790415077285126 us; error 41 us; skew 500 ppm
I20260926 09:31:17.286767 20243 webserver.cc:460] Webserver started at http://127.0.98.2:45597/ using document root <none> and password file <none>
I20260926 09:31:17.287056 20243 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:17.287135 20243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:17.287293 20243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:17.288367 20243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-1/data/instance:
uuid: "9ac62197d26449d6905d6edd00b58c06"
format_stamp: "Formatted at 2026-09-26 09:31:17 on dist-test-slave-dfzm"
server_key: "5dce136a079ad0616b68327fd4ef37f1"
server_key_iv: "08098786fec244ac0755ff293206bb01"
server_key_version: "encryptionkey@0"
I20260926 09:31:17.288760 20243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9ac62197d26449d6905d6edd00b58c06"
format_stamp: "Formatted at 2026-09-26 09:31:17 on dist-test-slave-dfzm"
server_key: "5dce136a079ad0616b68327fd4ef37f1"
server_key_iv: "08098786fec244ac0755ff293206bb01"
server_key_version: "encryptionkey@0"
I20260926 09:31:17.290174 20243 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:31:17.290946 20257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:17.291126 20243 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:31:17.291226 20243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-1/wal
uuid: "9ac62197d26449d6905d6edd00b58c06"
format_stamp: "Formatted at 2026-09-26 09:31:17 on dist-test-slave-dfzm"
server_key: "5dce136a079ad0616b68327fd4ef37f1"
server_key_iv: "08098786fec244ac0755ff293206bb01"
server_key_version: "encryptionkey@0"
I20260926 09:31:17.291308 20243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:17.308969 20243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:17.309300 20243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:17.309451 20243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:17.309788 20243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:31:17.310127 20243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:31:17.310182 20243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:17.310223 20243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:31:17.310251 20243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:17.316048 20243 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38145
I20260926 09:31:17.316123 20370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38145 every 8 connection(s)
I20260926 09:31:17.316443 20243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-1/data/info.pb
I20260926 09:31:17.317478 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 20243
I20260926 09:31:17.317632 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-1/wal/instance
I20260926 09:31:17.317853 392 external_mini_cluster.cc:1428] Setting key 77e439402db0fa4b41421855fec51ddb
I20260926 09:31:17.318833 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44025
--builtin_ntp_servers=127.0.98.20:42921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 09:31:17.322443 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:17.317271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48643 (local address 127.0.98.62:44025)
0926 09:31:17.317360 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:17.317363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:17.317637 (+ 274us) server_negotiation.cc:407] Connection header received
0926 09:31:17.318363 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:17.318364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:17.318377 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:17.318436 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:17.319344 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.319852 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.320910 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.321151 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.321209 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:17.322158 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:17.322159 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:17.322160 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:17.322165 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:17.322184 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:17.322189 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:17.322254 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:17.322361 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 09:31:17.322383 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 09:31:17.322749 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:17.317230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44025 (local address 127.0.98.2:48643)
0926 09:31:17.317435 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:17.317484 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:17.318260 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:17.318513 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:17.318517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:17.318527 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:17.319249 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:17.319256 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.319956 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.319957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:17.320830 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:17.320832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.321243 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.321244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:17.321404 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:17.321984 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:31:17.321985 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:17.322022 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:31:17.322050 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:17.322267 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:17.322275 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:17.322278 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:31:17.322360 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 09:31:17.322477 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260926 09:31:17.323311 20371 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44025
I20260926 09:31:17.323417 20371 heartbeater.cc:464] Registering TS with master...
I20260926 09:31:17.323629 20371 heartbeater.cc:511] Master 127.0.98.62:44025 requested a full tablet report, sending...
I20260926 09:31:17.324008 20051 ts_manager.cc:194] Registered new tserver with Master: 9ac62197d26449d6905d6edd00b58c06 (127.0.98.2:38145)
I20260926 09:31:17.324404 20051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:48643
W20260926 09:31:17.401203 20374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:17.401402 20374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:17.401424 20374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:17.402601 20374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:17.402639 20374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:17.402665 20374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 09:31:17.404017 20374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:17.404274 20374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:17.404654 20374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:17.409723 20380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:17.409754 20382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:17.409729 20379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:17.409904 20374 server_base.cc:1034] running on GCE node
I20260926 09:31:17.410117 20374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:17.410364 20374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:17.411553 20374 hybrid_clock.cc:648] HybridClock initialized: now 1790415077411538 us; error 29 us; skew 500 ppm
I20260926 09:31:17.413199 20374 webserver.cc:460] Webserver started at http://127.0.98.3:43459/ using document root <none> and password file <none>
I20260926 09:31:17.413726 20374 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:17.413914 20374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:17.414083 20374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:17.415177 20374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-2/data/instance:
uuid: "4d6af5fd66cc485b819a26b61502af44"
format_stamp: "Formatted at 2026-09-26 09:31:17 on dist-test-slave-dfzm"
server_key: "08c74043a81e763645e12d37a9b60f14"
server_key_iv: "ae0b0373f949a92bcf4773a1223f6534"
server_key_version: "encryptionkey@0"
I20260926 09:31:17.415539 20374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4d6af5fd66cc485b819a26b61502af44"
format_stamp: "Formatted at 2026-09-26 09:31:17 on dist-test-slave-dfzm"
server_key: "08c74043a81e763645e12d37a9b60f14"
server_key_iv: "ae0b0373f949a92bcf4773a1223f6534"
server_key_version: "encryptionkey@0"
I20260926 09:31:17.417088 20374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:31:17.417902 20388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:17.418092 20374 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:17.418192 20374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-2/wal
uuid: "4d6af5fd66cc485b819a26b61502af44"
format_stamp: "Formatted at 2026-09-26 09:31:17 on dist-test-slave-dfzm"
server_key: "08c74043a81e763645e12d37a9b60f14"
server_key_iv: "ae0b0373f949a92bcf4773a1223f6534"
server_key_version: "encryptionkey@0"
I20260926 09:31:17.418280 20374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:17.440784 20374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:17.441144 20374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:17.441315 20374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:17.441562 20374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:31:17.442029 20374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:31:17.442085 20374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:17.442121 20374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:31:17.442149 20374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:17.448274 20374 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46203
I20260926 09:31:17.448315 20501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46203 every 8 connection(s)
I20260926 09:31:17.448709 20374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-2/data/info.pb
I20260926 09:31:17.454207 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 20374
I20260926 09:31:17.454284 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:17.449605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51741 (local address 127.0.98.62:44025)
0926 09:31:17.449656 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:17.449658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:17.449900 (+ 242us) server_negotiation.cc:407] Connection header received
0926 09:31:17.450514 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:17.450516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:17.450527 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:17.450582 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:17.451246 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.451726 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.452750 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.452939 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.452997 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:17.453926 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:17.453927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:17.453927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:17.453932 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:17.453946 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:17.453949 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:17.454042 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:17.454142 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 09:31:17.454179 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:31:17.454324 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-2/wal/instance
I20260926 09:31:17.454502 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:17.449524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44025 (local address 127.0.98.3:51741)
0926 09:31:17.449755 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:17.449791 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:17.450388 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:17.450643 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:17.450647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:17.450655 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:17.451165 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:17.451170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.451827 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.451828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:17.452655 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:17.452657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.453038 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.453039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:17.453164 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:17.453788 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:31:17.453789 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:17.453819 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:31:17.453842 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:17.454047 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:17.454052 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:17.454054 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:31:17.454109 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 09:31:17.454192 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":117,"threads_started":1}
I20260926 09:31:17.454515 392 external_mini_cluster.cc:1428] Setting key 22ed6a6982345c1c6fcb071d839c253e
I20260926 09:31:17.454932 20502 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44025
I20260926 09:31:17.455060 20502 heartbeater.cc:464] Registering TS with master...
I20260926 09:31:17.455282 20502 heartbeater.cc:511] Master 127.0.98.62:44025 requested a full tablet report, sending...
I20260926 09:31:17.455638 20051 ts_manager.cc:194] Registered new tserver with Master: 4d6af5fd66cc485b819a26b61502af44 (127.0.98.3:46203)
I20260926 09:31:17.455967 20051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:51741
I20260926 09:31:17.466665 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:31:17.471460 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:17.468183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53610 (local address 127.0.98.62:44025)
0926 09:31:17.468286 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:17.468288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:17.468401 (+ 113us) server_negotiation.cc:407] Connection header received
0926 09:31:17.468515 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:17.468515 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:17.468530 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:17.468577 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:17.468779 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.469284 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.470143 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.470357 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.470447 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:17.471185 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:17.471186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:17.471187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:17.471191 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:17.471207 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:17.471211 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:17.471259 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:17.471339 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 09:31:17.471366 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 09:31:17.473358 20051 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:31:17.475229 20051 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:31:17.481753 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:17.477374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36033 (local address 127.0.0.1:56314)
0926 09:31:17.477896 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:17.477905 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:17.477980 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:17.478284 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:17.478284 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:17.478286 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:17.478342 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:17.478344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.479189 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.479190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:17.480789 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:17.480791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.481490 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.481491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:17.481535 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:17.481538 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:31:17.481566 (+ 28us) client_negotiation.cc:240] Negotiation successful
0926 09:31:17.481642 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":46,"thread_start_us":140,"threads_started":1}
I20260926 09:31:17.481786 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:17.477663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46548 (local address 127.0.98.2:38145)
0926 09:31:17.478105 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:17.478109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:17.478119 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:31:17.478157 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:17.478160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:17.478175 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:17.478386 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:17.478456 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.479475 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.480549 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.481309 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.481390 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:17.481491 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:17.481649 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 09:31:17.481696 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":85,"threads_started":1}
I20260926 09:31:17.481760 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:17.477356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38145 (local address 127.0.0.1:46548)
0926 09:31:17.477754 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:17.477765 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:17.477819 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:17.478273 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:17.478274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:17.478276 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:17.478316 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:17.478317 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.479704 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.479705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:17.480444 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:17.480446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.481412 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.481413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:17.481504 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:17.481519 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 09:31:17.481608 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 09:31:17.481642 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
I20260926 09:31:17.482091 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:17.477490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57178 (local address 127.0.98.3:46203)
0926 09:31:17.477667 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:17.477672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:17.477686 (+ 14us) server_negotiation.cc:407] Connection header received
0926 09:31:17.477793 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:17.477796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:17.477814 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:17.477898 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:17.478115 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.479247 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.480662 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.481326 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.481390 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:17.481479 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:17.481794 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 09:31:17.482032 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
I20260926 09:31:17.482298 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:17.477356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46203 (local address 127.0.0.1:57178)
0926 09:31:17.477568 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:17.477628 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:17.477696 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:17.477926 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:17.477930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:17.477933 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:17.478010 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:17.478011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.479334 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.479335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:17.480580 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:17.480583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.481517 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.481518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:17.481622 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:17.481627 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:31:17.481677 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 09:31:17.481716 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":27,"thread_start_us":88,"threads_started":1}
I20260926 09:31:17.482293 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:17.477842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56314 (local address 127.0.98.1:36033)
0926 09:31:17.478107 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:17.478112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:17.478122 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:31:17.478160 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:17.478162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:17.478177 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:17.478280 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:17.478400 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.479099 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.480868 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.481393 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.481978 (+ 585us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:17.482077 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:17.482102 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 09:31:17.482151 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":126,"threads_started":1}
I20260926 09:31:17.482829 20305 tablet_service.cc:1467] Processing CreateTablet for tablet 88f2a7c2e7254deb9ffe339d539ad885 (DEFAULT_TABLE table=test-table [id=eb5af022ed214b9dbb37d92d37c64f7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:31:17.482893 20436 tablet_service.cc:1467] Processing CreateTablet for tablet 88f2a7c2e7254deb9ffe339d539ad885 (DEFAULT_TABLE table=test-table [id=eb5af022ed214b9dbb37d92d37c64f7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:31:17.482909 20174 tablet_service.cc:1467] Processing CreateTablet for tablet 88f2a7c2e7254deb9ffe339d539ad885 (DEFAULT_TABLE table=test-table [id=eb5af022ed214b9dbb37d92d37c64f7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:31:17.483163 20436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f2a7c2e7254deb9ffe339d539ad885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:17.483186 20174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f2a7c2e7254deb9ffe339d539ad885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:17.483186 20305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f2a7c2e7254deb9ffe339d539ad885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:17.485742 20516 tablet_bootstrap.cc:492] T 88f2a7c2e7254deb9ffe339d539ad885 P 4d6af5fd66cc485b819a26b61502af44: Bootstrap starting.
I20260926 09:31:17.486399 20516 tablet_bootstrap.cc:654] T 88f2a7c2e7254deb9ffe339d539ad885 P 4d6af5fd66cc485b819a26b61502af44: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:17.486433 20518 tablet_bootstrap.cc:492] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677: Bootstrap starting.
I20260926 09:31:17.486732 20516 log.cc:824] T 88f2a7c2e7254deb9ffe339d539ad885 P 4d6af5fd66cc485b819a26b61502af44: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:17.487134 20518 tablet_bootstrap.cc:654] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:17.487461 20518 log.cc:824] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:17.487504 20516 tablet_bootstrap.cc:492] T 88f2a7c2e7254deb9ffe339d539ad885 P 4d6af5fd66cc485b819a26b61502af44: No bootstrap required, opened a new log
I20260926 09:31:17.487561 20516 ts_tablet_manager.cc:1397] T 88f2a7c2e7254deb9ffe339d539ad885 P 4d6af5fd66cc485b819a26b61502af44: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:31:17.487802 20517 tablet_bootstrap.cc:492] T 88f2a7c2e7254deb9ffe339d539ad885 P 9ac62197d26449d6905d6edd00b58c06: Bootstrap starting.
I20260926 09:31:17.488173 20518 tablet_bootstrap.cc:492] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677: No bootstrap required, opened a new log
I20260926 09:31:17.488256 20518 ts_tablet_manager.cc:1397] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:31:17.488483 20517 tablet_bootstrap.cc:654] T 88f2a7c2e7254deb9ffe339d539ad885 P 9ac62197d26449d6905d6edd00b58c06: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:17.488894 20517 log.cc:824] T 88f2a7c2e7254deb9ffe339d539ad885 P 9ac62197d26449d6905d6edd00b58c06: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:17.489673 20517 tablet_bootstrap.cc:492] T 88f2a7c2e7254deb9ffe339d539ad885 P 9ac62197d26449d6905d6edd00b58c06: No bootstrap required, opened a new log
I20260926 09:31:17.489746 20517 ts_tablet_manager.cc:1397] T 88f2a7c2e7254deb9ffe339d539ad885 P 9ac62197d26449d6905d6edd00b58c06: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:31:17.490527 20518 raft_consensus.cc:348] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac62197d26449d6905d6edd00b58c06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38145 } } peers { permanent_uuid: "4d6af5fd66cc485b819a26b61502af44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "f3165536400e4eab9bee7d92d885b677" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36033 } }
I20260926 09:31:17.490689 20518 raft_consensus.cc:374] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:17.490581 20516 raft_consensus.cc:348] T 88f2a7c2e7254deb9ffe339d539ad885 P 4d6af5fd66cc485b819a26b61502af44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac62197d26449d6905d6edd00b58c06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38145 } } peers { permanent_uuid: "4d6af5fd66cc485b819a26b61502af44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "f3165536400e4eab9bee7d92d885b677" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36033 } }
I20260926 09:31:17.490734 20518 raft_consensus.cc:729] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3165536400e4eab9bee7d92d885b677, State: Initialized, Role: FOLLOWER
I20260926 09:31:17.490737 20516 raft_consensus.cc:374] T 88f2a7c2e7254deb9ffe339d539ad885 P 4d6af5fd66cc485b819a26b61502af44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:17.490768 20516 raft_consensus.cc:729] T 88f2a7c2e7254deb9ffe339d539ad885 P 4d6af5fd66cc485b819a26b61502af44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d6af5fd66cc485b819a26b61502af44, State: Initialized, Role: FOLLOWER
I20260926 09:31:17.490865 20518 consensus_queue.cc:260] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac62197d26449d6905d6edd00b58c06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38145 } } peers { permanent_uuid: "4d6af5fd66cc485b819a26b61502af44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "f3165536400e4eab9bee7d92d885b677" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36033 } }
I20260926 09:31:17.490921 20516 consensus_queue.cc:260] T 88f2a7c2e7254deb9ffe339d539ad885 P 4d6af5fd66cc485b819a26b61502af44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac62197d26449d6905d6edd00b58c06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38145 } } peers { permanent_uuid: "4d6af5fd66cc485b819a26b61502af44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "f3165536400e4eab9bee7d92d885b677" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36033 } }
I20260926 09:31:17.491173 20516 ts_tablet_manager.cc:1428] T 88f2a7c2e7254deb9ffe339d539ad885 P 4d6af5fd66cc485b819a26b61502af44: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 09:31:17.491207 20502 heartbeater.cc:503] Master 127.0.98.62:44025 was elected leader, sending a full tablet report...
I20260926 09:31:17.491150 20518 ts_tablet_manager.cc:1428] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:31:17.491560 20240 heartbeater.cc:503] Master 127.0.98.62:44025 was elected leader, sending a full tablet report...
I20260926 09:31:17.492380 20517 raft_consensus.cc:348] T 88f2a7c2e7254deb9ffe339d539ad885 P 9ac62197d26449d6905d6edd00b58c06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac62197d26449d6905d6edd00b58c06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38145 } } peers { permanent_uuid: "4d6af5fd66cc485b819a26b61502af44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "f3165536400e4eab9bee7d92d885b677" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36033 } }
I20260926 09:31:17.492569 20517 raft_consensus.cc:374] T 88f2a7c2e7254deb9ffe339d539ad885 P 9ac62197d26449d6905d6edd00b58c06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:17.492621 20517 raft_consensus.cc:729] T 88f2a7c2e7254deb9ffe339d539ad885 P 9ac62197d26449d6905d6edd00b58c06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ac62197d26449d6905d6edd00b58c06, State: Initialized, Role: FOLLOWER
I20260926 09:31:17.492799 20517 consensus_queue.cc:260] T 88f2a7c2e7254deb9ffe339d539ad885 P 9ac62197d26449d6905d6edd00b58c06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac62197d26449d6905d6edd00b58c06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38145 } } peers { permanent_uuid: "4d6af5fd66cc485b819a26b61502af44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "f3165536400e4eab9bee7d92d885b677" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36033 } }
I20260926 09:31:17.493101 20517 ts_tablet_manager.cc:1428] T 88f2a7c2e7254deb9ffe339d539ad885 P 9ac62197d26449d6905d6edd00b58c06: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:31:17.493139 20371 heartbeater.cc:503] Master 127.0.98.62:44025 was elected leader, sending a full tablet report...
I20260926 09:31:17.521180 20522 raft_consensus.cc:482] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:31:17.521307 20522 raft_consensus.cc:504] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac62197d26449d6905d6edd00b58c06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38145 } } peers { permanent_uuid: "4d6af5fd66cc485b819a26b61502af44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "f3165536400e4eab9bee7d92d885b677" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36033 } }
I20260926 09:31:17.521811 20522 leader_election.cc:288] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ac62197d26449d6905d6edd00b58c06 (127.0.98.2:38145), 4d6af5fd66cc485b819a26b61502af44 (127.0.98.3:46203)
I20260926 09:31:17.524972 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:17.522226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58879 (local address 127.0.98.2:38145)
0926 09:31:17.522272 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:17.522274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:17.522285 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:31:17.522337 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:17.522338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:17.522349 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:17.522413 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:17.522593 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.523251 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.524200 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.524592 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.524689 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:17.524698 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:17.524872 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 09:31:17.524909 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:31:17.524996 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:17.522037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38145 (local address 127.0.98.1:58879)
0926 09:31:17.522205 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:17.522216 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:17.522280 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:17.522413 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:17.522413 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:17.522415 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:17.522501 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:17.522502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.523327 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.523328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:17.524086 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:17.524088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.524694 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.524695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:17.524781 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:17.524785 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:31:17.524864 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 09:31:17.524902 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":2,"thread_start_us":94,"threads_started":1}
I20260926 09:31:17.525274 20325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f2a7c2e7254deb9ffe339d539ad885" candidate_uuid: "f3165536400e4eab9bee7d92d885b677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac62197d26449d6905d6edd00b58c06" is_pre_election: true
I20260926 09:31:17.525403 20325 raft_consensus.cc:2454] T 88f2a7c2e7254deb9ffe339d539ad885 P 9ac62197d26449d6905d6edd00b58c06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3165536400e4eab9bee7d92d885b677 in term 0.
I20260926 09:31:17.525635 20242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:17.522025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46203 (local address 127.0.98.1:45323)
0926 09:31:17.522109 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:17.522121 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:17.522167 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:17.522327 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:17.522328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:17.522330 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:31:17.522404 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:17.522406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.523266 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.523267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:17.524020 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:17.524022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.525343 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.525344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:17.525437 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:17.525440 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:31:17.525494 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 09:31:17.525538 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 09:31:17.525700 20130 leader_election.cc:302] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ac62197d26449d6905d6edd00b58c06, f3165536400e4eab9bee7d92d885b677; no voters:
I20260926 09:31:17.525823 20522 raft_consensus.cc:2781] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:31:17.525861 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:17.522093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45323 (local address 127.0.98.3:46203)
0926 09:31:17.522153 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:17.522155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:17.522167 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:31:17.522238 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:17.522239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:17.522249 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:17.522310 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:31:17.522480 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.523197 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.524614 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.524953 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.525051 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:17.525061 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:17.525735 (+ 674us) server_negotiation.cc:299] Negotiation successful
0926 09:31:17.525785 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 09:31:17.525878 20522 raft_consensus.cc:482] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:31:17.525902 20522 raft_consensus.cc:3037] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:17.526129 20456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f2a7c2e7254deb9ffe339d539ad885" candidate_uuid: "f3165536400e4eab9bee7d92d885b677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d6af5fd66cc485b819a26b61502af44" is_pre_election: true
I20260926 09:31:17.526238 20456 raft_consensus.cc:2454] T 88f2a7c2e7254deb9ffe339d539ad885 P 4d6af5fd66cc485b819a26b61502af44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3165536400e4eab9bee7d92d885b677 in term 0.
I20260926 09:31:17.526777 20522 raft_consensus.cc:504] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac62197d26449d6905d6edd00b58c06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38145 } } peers { permanent_uuid: "4d6af5fd66cc485b819a26b61502af44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "f3165536400e4eab9bee7d92d885b677" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36033 } }
I20260926 09:31:17.526933 20522 leader_election.cc:288] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677 [CANDIDATE]: Term 1 election: Requested vote from peers 9ac62197d26449d6905d6edd00b58c06 (127.0.98.2:38145), 4d6af5fd66cc485b819a26b61502af44 (127.0.98.3:46203)
I20260926 09:31:17.527091 20325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f2a7c2e7254deb9ffe339d539ad885" candidate_uuid: "f3165536400e4eab9bee7d92d885b677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac62197d26449d6905d6edd00b58c06"
I20260926 09:31:17.527175 20325 raft_consensus.cc:3037] T 88f2a7c2e7254deb9ffe339d539ad885 P 9ac62197d26449d6905d6edd00b58c06 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:17.527153 20456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f2a7c2e7254deb9ffe339d539ad885" candidate_uuid: "f3165536400e4eab9bee7d92d885b677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d6af5fd66cc485b819a26b61502af44"
I20260926 09:31:17.527243 20456 raft_consensus.cc:3037] T 88f2a7c2e7254deb9ffe339d539ad885 P 4d6af5fd66cc485b819a26b61502af44 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:17.527877 20325 raft_consensus.cc:2454] T 88f2a7c2e7254deb9ffe339d539ad885 P 9ac62197d26449d6905d6edd00b58c06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3165536400e4eab9bee7d92d885b677 in term 1.
I20260926 09:31:17.528036 20456 raft_consensus.cc:2454] T 88f2a7c2e7254deb9ffe339d539ad885 P 4d6af5fd66cc485b819a26b61502af44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3165536400e4eab9bee7d92d885b677 in term 1.
I20260926 09:31:17.528065 20130 leader_election.cc:302] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ac62197d26449d6905d6edd00b58c06, f3165536400e4eab9bee7d92d885b677; no voters:
I20260926 09:31:17.528141 20522 raft_consensus.cc:2781] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:17.528308 20522 raft_consensus.cc:686] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677 [term 1 LEADER]: Becoming Leader. State: Replica: f3165536400e4eab9bee7d92d885b677, State: Running, Role: LEADER
I20260926 09:31:17.528432 20522 consensus_queue.cc:237] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac62197d26449d6905d6edd00b58c06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38145 } } peers { permanent_uuid: "4d6af5fd66cc485b819a26b61502af44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } } peers { permanent_uuid: "f3165536400e4eab9bee7d92d885b677" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36033 } }
I20260926 09:31:17.529348 20051 catalog_manager.cc:5495] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3165536400e4eab9bee7d92d885b677 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f3165536400e4eab9bee7d92d885b677" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac62197d26449d6905d6edd00b58c06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d6af5fd66cc485b819a26b61502af44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3165536400e4eab9bee7d92d885b677" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36033 } health_report { overall_health: HEALTHY } } }
I20260926 09:31:17.540493 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:17.538192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56322 (local address 127.0.98.1:36033)
0926 09:31:17.538237 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:17.538239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:17.538247 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:31:17.538282 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:17.538283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:17.538302 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:17.538413 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:17.538541 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.538992 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.539727 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:17.539933 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:17.539996 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:17.540141 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:17.540234 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:17.540299 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:17.540383 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 09:31:17.540406 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:31:17.542613 20456 raft_consensus.cc:1261] T 88f2a7c2e7254deb9ffe339d539ad885 P 4d6af5fd66cc485b819a26b61502af44 [term 1 FOLLOWER]: Refusing update from remote peer f3165536400e4eab9bee7d92d885b677: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:31:17.542613 20325 raft_consensus.cc:1261] T 88f2a7c2e7254deb9ffe339d539ad885 P 9ac62197d26449d6905d6edd00b58c06 [term 1 FOLLOWER]: Refusing update from remote peer f3165536400e4eab9bee7d92d885b677: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:31:17.542999 20526 consensus_queue.cc:1035] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac62197d26449d6905d6edd00b58c06" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:31:17.543159 20522 consensus_queue.cc:1035] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d6af5fd66cc485b819a26b61502af44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:31:17.545909 20530 mvcc.cc:205] Tried to move back new op lower bound from 7333540157611995136 to 7333540157557911552. Current Snapshot: MvccSnapshot[applied={T|T < 7333540157611995136}]
I20260926 09:31:17.546069 20531 mvcc.cc:205] Tried to move back new op lower bound from 7333540157611995136 to 7333540157557911552. Current Snapshot: MvccSnapshot[applied={T|T < 7333540157611995136}]
I20260926 09:31:17.548316 20052 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:31:17.548422 20052 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:31:17.549139 20052 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb71d581c2504fdf8d9d80faabdec7c2: Sending DeleteTablet for 3 replicas of tablet 88f2a7c2e7254deb9ffe339d539ad885
I20260926 09:31:17.549530 20174 tablet_service.cc:1514] Processing DeleteTablet for tablet 88f2a7c2e7254deb9ffe339d539ad885 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:31:17 UTC) from {username='slave'} at 127.0.0.1:56314
I20260926 09:31:17.549672 20305 tablet_service.cc:1514] Processing DeleteTablet for tablet 88f2a7c2e7254deb9ffe339d539ad885 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:31:17 UTC) from {username='slave'} at 127.0.0.1:46548
I20260926 09:31:17.549818 20539 tablet_replica.cc:330] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677: stopping tablet replica
I20260926 09:31:17.549536 20436 tablet_service.cc:1514] Processing DeleteTablet for tablet 88f2a7c2e7254deb9ffe339d539ad885 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:31:17 UTC) from {username='slave'} at 127.0.0.1:57178
I20260926 09:31:17.549924 20539 raft_consensus.cc:2229] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:31:17.550045 20539 raft_consensus.cc:2258] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:31:17.550056 20541 tablet_replica.cc:330] T 88f2a7c2e7254deb9ffe339d539ad885 P 4d6af5fd66cc485b819a26b61502af44: stopping tablet replica
I20260926 09:31:17.550243 20541 raft_consensus.cc:2229] T 88f2a7c2e7254deb9ffe339d539ad885 P 4d6af5fd66cc485b819a26b61502af44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:31:17.550436 20541 raft_consensus.cc:2258] T 88f2a7c2e7254deb9ffe339d539ad885 P 4d6af5fd66cc485b819a26b61502af44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:31:17.550428 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 20112
I20260926 09:31:17.550498 20539 ts_tablet_manager.cc:1905] T 88f2a7c2e7254deb9ffe339d539ad885 P f3165536400e4eab9bee7d92d885b677: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:31:17.551457 20541 ts_tablet_manager.cc:1905] T 88f2a7c2e7254deb9ffe339d539ad885 P 4d6af5fd66cc485b819a26b61502af44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:31:17.553378 20541 ts_tablet_manager.cc:1918] T 88f2a7c2e7254deb9ffe339d539ad885 P 4d6af5fd66cc485b819a26b61502af44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:31:17.553455 20541 log.cc:1196] T 88f2a7c2e7254deb9ffe339d539ad885 P 4d6af5fd66cc485b819a26b61502af44: Deleting WAL directory at /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414839069479-392-0/minicluster-data/ts-2/wal/wals/88f2a7c2e7254deb9ffe339d539ad885
I20260926 09:31:17.553730 20541 ts_tablet_manager.cc:1939] T 88f2a7c2e7254deb9ffe339d539ad885 P 4d6af5fd66cc485b819a26b61502af44: Deleting consensus metadata
I20260926 09:31:17.554168 20038 catalog_manager.cc:4841] TS 4d6af5fd66cc485b819a26b61502af44 (127.0.98.3:46203): tablet 88f2a7c2e7254deb9ffe339d539ad885 (table test-table [id=eb5af022ed214b9dbb37d92d37c64f7f]) successfully deleted
W20260926 09:31:17.557226 20036 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36033 (error 108)
I20260926 09:31:17.557466 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 20243
I20260926 09:31:17.557502 20540 tablet_replica.cc:330] T 88f2a7c2e7254deb9ffe339d539ad885 P 9ac62197d26449d6905d6edd00b58c06: stopping tablet replica
I20260926 09:31:17.557636 20540 raft_consensus.cc:2229] T 88f2a7c2e7254deb9ffe339d539ad885 P 9ac62197d26449d6905d6edd00b58c06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:31:17.557747 20540 raft_consensus.cc:2258] T 88f2a7c2e7254deb9ffe339d539ad885 P 9ac62197d26449d6905d6edd00b58c06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:31:17.558208 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:17.557691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36033 (local address 127.0.0.1:56326)
0926 09:31:17.558129 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:17.558170 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417}
W20260926 09:31:17.558454 20036 catalog_manager.cc:4584] TS f3165536400e4eab9bee7d92d885b677 (127.0.98.1:36033): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 88f2a7c2e7254deb9ffe339d539ad885: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36033: connect: Connection refused (error 111)
I20260926 09:31:17.563143 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 20374
I20260926 09:31:17.563184 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:17.563015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38145 (local address 127.0.0.1:46558)
0926 09:31:17.563101 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:17.563142 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 09:31:17.567867 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 20021
2026-09-26T09:31:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (702 ms)
[----------] 6 tests from EncryptionPolicyTest (8588 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:31:17 dist-test-slave-dfzm krb5kdc[20545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:31:17 dist-test-slave-dfzm krb5kdc[20545](info): set up 2 sockets
Sep 26 09:31:17 dist-test-slave-dfzm krb5kdc[20545](info): commencing operation
krb5kdc: starting...
W20260926 09:31:18.609561 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:31:18 dist-test-slave-dfzm krb5kdc[20545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:31:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:31:18.729475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:18.813004 20558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:18.813215 20558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:18.813237 20558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:18.814440 20558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:18.814487 20558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:18.814502 20558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:18.814514 20558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:18.814527 20558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:18.815938 20558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36973
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:18.816133 20558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:18.816301 20558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:18.821172 20566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:18.821231 20563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:18.821234 20564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:18.821506 20558 server_base.cc:1034] running on GCE node
I20260926 09:31:18.821782 20558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:18.822074 20558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:18.823232 20558 hybrid_clock.cc:648] HybridClock initialized: now 1790415078823219 us; error 30 us; skew 500 ppm
Sep 26 09:31:18 dist-test-slave-dfzm krb5kdc[20545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:18.826049 20558 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:31:18.826539 20558 webserver.cc:460] Webserver started at http://127.0.0.1:41583/ using document root <none> and password file <none>
I20260926 09:31:18.826783 20558 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:18.826851 20558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:18.827001 20558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:18.828111 20558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "2deca4cb24194d89b404b80f3df43df8"
format_stamp: "Formatted at 2026-09-26 09:31:18 on dist-test-slave-dfzm"
server_key: "8da948f939ce1bbc5232ab1a07d23994"
server_key_iv: "101eb36465846d6e8a83f1759b2b0815"
server_key_version: "encryptionkey@0"
I20260926 09:31:18.828473 20558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "2deca4cb24194d89b404b80f3df43df8"
format_stamp: "Formatted at 2026-09-26 09:31:18 on dist-test-slave-dfzm"
server_key: "8da948f939ce1bbc5232ab1a07d23994"
server_key_iv: "101eb36465846d6e8a83f1759b2b0815"
server_key_version: "encryptionkey@0"
I20260926 09:31:18.829818 20558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:31:18.830591 20573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:18.830781 20558 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:18.830873 20558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "2deca4cb24194d89b404b80f3df43df8"
format_stamp: "Formatted at 2026-09-26 09:31:18 on dist-test-slave-dfzm"
server_key: "8da948f939ce1bbc5232ab1a07d23994"
server_key_iv: "101eb36465846d6e8a83f1759b2b0815"
server_key_version: "encryptionkey@0"
I20260926 09:31:18.830951 20558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:18.842275 20558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:18.844548 20558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:18.844672 20558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:18.848538 20558 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36973
I20260926 09:31:18.848577 20625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36973 every 8 connection(s)
I20260926 09:31:18.848948 20558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:18.849958 20626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:18.852306 20626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2deca4cb24194d89b404b80f3df43df8: Bootstrap starting.
I20260926 09:31:18.852905 20626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2deca4cb24194d89b404b80f3df43df8: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:18.853214 20626 log.cc:824] T 00000000000000000000000000000000 P 2deca4cb24194d89b404b80f3df43df8: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:18.854106 20626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2deca4cb24194d89b404b80f3df43df8: No bootstrap required, opened a new log
I20260926 09:31:18.854421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 20558
I20260926 09:31:18.854552 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:18.854728 392 external_mini_cluster.cc:1428] Setting key a78362d313e43196781881302df813be
I20260926 09:31:18.856464 20626 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2deca4cb24194d89b404b80f3df43df8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2deca4cb24194d89b404b80f3df43df8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } }
I20260926 09:31:18.856593 20626 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2deca4cb24194d89b404b80f3df43df8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:18.856616 20626 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2deca4cb24194d89b404b80f3df43df8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2deca4cb24194d89b404b80f3df43df8, State: Initialized, Role: FOLLOWER
I20260926 09:31:18.856753 20626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2deca4cb24194d89b404b80f3df43df8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2deca4cb24194d89b404b80f3df43df8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } }
I20260926 09:31:18.856814 20626 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2deca4cb24194d89b404b80f3df43df8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:31:18.856835 20626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2deca4cb24194d89b404b80f3df43df8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:31:18.856910 20626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2deca4cb24194d89b404b80f3df43df8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:18.857824 20626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2deca4cb24194d89b404b80f3df43df8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2deca4cb24194d89b404b80f3df43df8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } }
I20260926 09:31:18.857952 20626 leader_election.cc:302] T 00000000000000000000000000000000 P 2deca4cb24194d89b404b80f3df43df8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2deca4cb24194d89b404b80f3df43df8; no voters:
I20260926 09:31:18.858122 20626 leader_election.cc:288] T 00000000000000000000000000000000 P 2deca4cb24194d89b404b80f3df43df8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:31:18.858242 20631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2deca4cb24194d89b404b80f3df43df8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:18.858359 20631 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2deca4cb24194d89b404b80f3df43df8 [term 1 LEADER]: Becoming Leader. State: Replica: 2deca4cb24194d89b404b80f3df43df8, State: Running, Role: LEADER
I20260926 09:31:18.858388 20626 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2deca4cb24194d89b404b80f3df43df8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:18.858453 20631 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2deca4cb24194d89b404b80f3df43df8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2deca4cb24194d89b404b80f3df43df8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } }
I20260926 09:31:18.858767 20631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2deca4cb24194d89b404b80f3df43df8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2deca4cb24194d89b404b80f3df43df8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2deca4cb24194d89b404b80f3df43df8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } } }
I20260926 09:31:18.858853 20631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2deca4cb24194d89b404b80f3df43df8 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:18.858978 20631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2deca4cb24194d89b404b80f3df43df8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2deca4cb24194d89b404b80f3df43df8. Latest consensus state: current_term: 1 leader_uuid: "2deca4cb24194d89b404b80f3df43df8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2deca4cb24194d89b404b80f3df43df8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } } }
I20260926 09:31:18.859047 20631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2deca4cb24194d89b404b80f3df43df8 [sys.catalog]: This master's current role is: LEADER
W20260926 09:31:18.860204 20647 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2deca4cb24194d89b404b80f3df43df8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:31:18.860275 20647 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:31:18.860327 20637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:18.860499 20637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 09:31:18 dist-test-slave-dfzm krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 09:31:18.862390 20637 catalog_manager.cc:1347] Generated new cluster ID: 84ac374a0aed4169ac726f34ccbe4123
I20260926 09:31:18.862439 20637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:31:18.866607 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:18.855248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53175 (local address 127.0.0.1:36973)
0926 09:31:18.855453 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:18.855463 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:18.855494 (+ 31us) server_negotiation.cc:407] Connection header received
0926 09:31:18.856045 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:18.856054 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:18.856129 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:18.856288 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:18.857051 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:18.858141 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:18.859252 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:18.860011 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:18.860132 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:18.862798 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:18.862802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:18.862802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:18.862814 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:18.864859 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:18.865286 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:18.865289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:18.865289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:18.865404 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:18.865658 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:18.865659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:18.865659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:18.866001 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:18.866181 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:18.866292 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 09:31:18.866406 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":80,"threads_started":1}
I20260926 09:31:18.867620 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:31:18.869493 20637 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:18.870042 20637 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:18.880007 20637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2deca4cb24194d89b404b80f3df43df8: Generated new TSK 0
I20260926 09:31:18.880203 20637 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:31:18.885704 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:18.869416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48611 (local address 127.0.0.1:36973)
0926 09:31:18.869511 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:18.869512 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:18.869520 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:31:18.869538 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:18.869539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:18.869547 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:18.869620 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:18.879100 (+ 9480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:18.880246 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:18.881317 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:18.881536 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:18.881660 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:18.883056 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:18.883057 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:18.883057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:18.883063 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:18.884427 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:18.884866 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:18.884867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:18.884868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:18.884926 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:18.885164 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:18.885165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:18.885165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:18.885266 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:18.885356 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:18.885499 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 09:31:18.885553 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 09:31:18.887586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 20558
W20260926 09:31:18.892046 20543 connection.cc:535] client connection to 127.0.0.1:36973 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36973): Connection reset by peer (error 104)
2026-09-26T09:31:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:31:18 dist-test-slave-dfzm krb5kdc[20657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:31:18 dist-test-slave-dfzm krb5kdc[20657](info): set up 2 sockets
Sep 26 09:31:18 dist-test-slave-dfzm krb5kdc[20657](info): commencing operation
krb5kdc: starting...
W20260926 09:31:19.921684 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:31:20 dist-test-slave-dfzm krb5kdc[20657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:31:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:31:20.038671 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:20.119578 20670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:20.119792 20670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:20.119815 20670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:20.120970 20670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:20.121011 20670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:20.121026 20670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:20.121078 20670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:20.121104 20670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:20.122577 20670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36257
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:20.122850 20670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:20.123126 20670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:20.127800 20676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:20.127853 20678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:20.127853 20675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:20.128162 20670 server_base.cc:1034] running on GCE node
I20260926 09:31:20.128443 20670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:20.128731 20670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:20.129890 20670 hybrid_clock.cc:648] HybridClock initialized: now 1790415080129866 us; error 39 us; skew 500 ppm
Sep 26 09:31:20 dist-test-slave-dfzm krb5kdc[20657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:20.138118 20670 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:31:20.138520 20670 webserver.cc:460] Webserver started at http://127.0.0.1:41693/ using document root <none> and password file <none>
I20260926 09:31:20.138700 20670 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:20.138741 20670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:20.138829 20670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:20.139842 20670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "0eb63e465ac5492c9d8fdabb206efd43"
format_stamp: "Formatted at 2026-09-26 09:31:20 on dist-test-slave-dfzm"
server_key: "58c6e60703e6420ef44684279f843736"
server_key_iv: "8a2eea4373b8d82da5b6058ad23f92d2"
server_key_version: "encryptionkey@0"
I20260926 09:31:20.140134 20670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "0eb63e465ac5492c9d8fdabb206efd43"
format_stamp: "Formatted at 2026-09-26 09:31:20 on dist-test-slave-dfzm"
server_key: "58c6e60703e6420ef44684279f843736"
server_key_iv: "8a2eea4373b8d82da5b6058ad23f92d2"
server_key_version: "encryptionkey@0"
I20260926 09:31:20.141424 20670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:31:20.142246 20685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:20.142443 20670 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:31:20.142530 20670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "0eb63e465ac5492c9d8fdabb206efd43"
format_stamp: "Formatted at 2026-09-26 09:31:20 on dist-test-slave-dfzm"
server_key: "58c6e60703e6420ef44684279f843736"
server_key_iv: "8a2eea4373b8d82da5b6058ad23f92d2"
server_key_version: "encryptionkey@0"
I20260926 09:31:20.142584 20670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:20.166435 20670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:20.168823 20670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:20.168942 20670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:20.172863 20670 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36257
I20260926 09:31:20.172914 20737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36257 every 8 connection(s)
I20260926 09:31:20.173269 20670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:20.173710 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 20670
I20260926 09:31:20.173847 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:20.174054 392 external_mini_cluster.cc:1428] Setting key 72eccc2d29cc6824de6cae0db5ae1d1c
I20260926 09:31:20.174382 20738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:20.177657 20738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eb63e465ac5492c9d8fdabb206efd43: Bootstrap starting.
I20260926 09:31:20.178272 20738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0eb63e465ac5492c9d8fdabb206efd43: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:20.178573 20738 log.cc:824] T 00000000000000000000000000000000 P 0eb63e465ac5492c9d8fdabb206efd43: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:20.179704 20738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eb63e465ac5492c9d8fdabb206efd43: No bootstrap required, opened a new log
I20260926 09:31:20.179931 20741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:20.174649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41105 (local address 127.0.0.1:36257)
0926 09:31:20.174828 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:20.174835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:20.174860 (+ 25us) server_negotiation.cc:407] Connection header received
0926 09:31:20.175508 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:20.175515 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:20.175563 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:20.175685 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:20.175916 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:20.176770 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:20.177902 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:20.178324 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:20.178459 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:20.179202 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:20.179207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:20.179207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:20.179417 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:20.179449 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:20.179461 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:20.179531 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:20.179648 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 09:31:20.179691 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260926 09:31:20.180850 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:31:20.182037 20738 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0eb63e465ac5492c9d8fdabb206efd43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb63e465ac5492c9d8fdabb206efd43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36257 } }
I20260926 09:31:20.182186 20738 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0eb63e465ac5492c9d8fdabb206efd43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:20.182231 20738 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0eb63e465ac5492c9d8fdabb206efd43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eb63e465ac5492c9d8fdabb206efd43, State: Initialized, Role: FOLLOWER
I20260926 09:31:20.182348 20738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eb63e465ac5492c9d8fdabb206efd43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb63e465ac5492c9d8fdabb206efd43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36257 } }
I20260926 09:31:20.182440 20738 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0eb63e465ac5492c9d8fdabb206efd43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:31:20.182487 20738 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0eb63e465ac5492c9d8fdabb206efd43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:31:20.182544 20738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0eb63e465ac5492c9d8fdabb206efd43 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:20.183092 20738 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0eb63e465ac5492c9d8fdabb206efd43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb63e465ac5492c9d8fdabb206efd43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36257 } }
I20260926 09:31:20.183175 20738 leader_election.cc:302] T 00000000000000000000000000000000 P 0eb63e465ac5492c9d8fdabb206efd43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0eb63e465ac5492c9d8fdabb206efd43; no voters:
I20260926 09:31:20.183372 20738 leader_election.cc:288] T 00000000000000000000000000000000 P 0eb63e465ac5492c9d8fdabb206efd43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:31:20.183399 20748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0eb63e465ac5492c9d8fdabb206efd43 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:20.183547 20748 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0eb63e465ac5492c9d8fdabb206efd43 [term 1 LEADER]: Becoming Leader. State: Replica: 0eb63e465ac5492c9d8fdabb206efd43, State: Running, Role: LEADER
I20260926 09:31:20.183630 20738 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0eb63e465ac5492c9d8fdabb206efd43 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:20.183665 20748 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0eb63e465ac5492c9d8fdabb206efd43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb63e465ac5492c9d8fdabb206efd43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36257 } }
I20260926 09:31:20.183991 20748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eb63e465ac5492c9d8fdabb206efd43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eb63e465ac5492c9d8fdabb206efd43. Latest consensus state: current_term: 1 leader_uuid: "0eb63e465ac5492c9d8fdabb206efd43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb63e465ac5492c9d8fdabb206efd43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36257 } } }
I20260926 09:31:20.184087 20748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eb63e465ac5492c9d8fdabb206efd43 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:20.184000 20749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eb63e465ac5492c9d8fdabb206efd43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0eb63e465ac5492c9d8fdabb206efd43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb63e465ac5492c9d8fdabb206efd43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36257 } } }
I20260926 09:31:20.184226 20749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eb63e465ac5492c9d8fdabb206efd43 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:20.184330 20755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:20.184840 20755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:20.185860 20741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:20.182230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56807 (local address 127.0.0.1:36257)
0926 09:31:20.182268 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:20.182270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:20.182429 (+ 159us) server_negotiation.cc:407] Connection header received
0926 09:31:20.182511 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:20.182512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:20.182522 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:20.182583 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:20.182791 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:20.183342 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:20.184373 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:20.184598 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:20.184679 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:20.185537 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:20.185538 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:20.185538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:20.185543 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:20.185556 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:20.185561 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:20.185625 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:20.185776 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 09:31:20.185787 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:31:20.187157 20755 catalog_manager.cc:1347] Generated new cluster ID: 8a35b159ba12489789978639bef4c3f3
I20260926 09:31:20.187209 20755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:31:20.204281 20755 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:20.205021 20755 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:20.211762 20755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0eb63e465ac5492c9d8fdabb206efd43: Generated new TSK 0
I20260926 09:31:20.211959 20755 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:31:20.217753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 20670
W20260926 09:31:20.222056 20655 connection.cc:535] client connection to 127.0.0.1:36257 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36257): Connection reset by peer (error 104)
2026-09-26T09:31:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:31:20 dist-test-slave-dfzm krb5kdc[20769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:31:20 dist-test-slave-dfzm krb5kdc[20769](info): set up 2 sockets
Sep 26 09:31:20 dist-test-slave-dfzm krb5kdc[20769](info): commencing operation
krb5kdc: starting...
W20260926 09:31:21.261647 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:31:21 dist-test-slave-dfzm krb5kdc[20769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:31:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:31:21.380965 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34251
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:21.462087 20782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:21.462353 20782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:21.462393 20782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:21.463603 20782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:21.463677 20782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:21.463703 20782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:21.463734 20782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:21.463758 20782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:21.465378 20782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34251
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:21.465689 20782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:21.465965 20782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:21.470970 20790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:21.470970 20787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:21.470958 20788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:21.471277 20782 server_base.cc:1034] running on GCE node
I20260926 09:31:21.471493 20782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:21.471779 20782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:21.472918 20782 hybrid_clock.cc:648] HybridClock initialized: now 1790415081472906 us; error 30 us; skew 500 ppm
Sep 26 09:31:21 dist-test-slave-dfzm krb5kdc[20769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:21.475574 20782 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:31:21.476070 20782 webserver.cc:460] Webserver started at http://127.0.0.1:34643/ using document root <none> and password file <none>
I20260926 09:31:21.476302 20782 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:21.476346 20782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:21.476490 20782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:21.477645 20782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "9eae3152a6a640e9ac6fcac8fccd537b"
format_stamp: "Formatted at 2026-09-26 09:31:21 on dist-test-slave-dfzm"
server_key: "f904fef0248871fccbb92b3b03d1cfcd"
server_key_iv: "4fd5f6172951856f64786355a804c951"
server_key_version: "encryptionkey@0"
I20260926 09:31:21.478022 20782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "9eae3152a6a640e9ac6fcac8fccd537b"
format_stamp: "Formatted at 2026-09-26 09:31:21 on dist-test-slave-dfzm"
server_key: "f904fef0248871fccbb92b3b03d1cfcd"
server_key_iv: "4fd5f6172951856f64786355a804c951"
server_key_version: "encryptionkey@0"
I20260926 09:31:21.479386 20782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:21.480224 20797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:21.480445 20782 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:21.480599 20782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "9eae3152a6a640e9ac6fcac8fccd537b"
format_stamp: "Formatted at 2026-09-26 09:31:21 on dist-test-slave-dfzm"
server_key: "f904fef0248871fccbb92b3b03d1cfcd"
server_key_iv: "4fd5f6172951856f64786355a804c951"
server_key_version: "encryptionkey@0"
I20260926 09:31:21.480683 20782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:21.484086 20782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:21.486405 20782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:21.486562 20782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:21.490423 20782 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34251
I20260926 09:31:21.490483 20849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34251 every 8 connection(s)
I20260926 09:31:21.490829 20782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:21.491806 20850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:21.494282 20850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eae3152a6a640e9ac6fcac8fccd537b: Bootstrap starting.
I20260926 09:31:21.494879 20850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9eae3152a6a640e9ac6fcac8fccd537b: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:21.495208 20850 log.cc:824] T 00000000000000000000000000000000 P 9eae3152a6a640e9ac6fcac8fccd537b: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:21.495716 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 20782
I20260926 09:31:21.495862 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:21.496080 392 external_mini_cluster.cc:1428] Setting key d32ed4da0ea25bd6e193011129fbe5e7
I20260926 09:31:21.496104 20850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eae3152a6a640e9ac6fcac8fccd537b: No bootstrap required, opened a new log
I20260926 09:31:21.498116 20850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9eae3152a6a640e9ac6fcac8fccd537b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eae3152a6a640e9ac6fcac8fccd537b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34251 } }
I20260926 09:31:21.498293 20850 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9eae3152a6a640e9ac6fcac8fccd537b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:21.498338 20850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9eae3152a6a640e9ac6fcac8fccd537b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eae3152a6a640e9ac6fcac8fccd537b, State: Initialized, Role: FOLLOWER
I20260926 09:31:21.498507 20850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9eae3152a6a640e9ac6fcac8fccd537b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eae3152a6a640e9ac6fcac8fccd537b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34251 } }
I20260926 09:31:21.498627 20850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9eae3152a6a640e9ac6fcac8fccd537b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:31:21.498693 20850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9eae3152a6a640e9ac6fcac8fccd537b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:31:21.498771 20850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9eae3152a6a640e9ac6fcac8fccd537b [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:21.499616 20854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:21.496616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56533 (local address 127.0.0.1:34251)
0926 09:31:21.496801 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:21.496808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:21.496835 (+ 27us) server_negotiation.cc:407] Connection header received
0926 09:31:21.498102 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:21.498110 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:21.498141 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:21.498269 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:21.499048 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:21.499052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:21.499052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:21.499057 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:21.499107 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:21.499124 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:21.499213 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:21.499279 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 09:31:21.499331 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260926 09:31:21.500022 20850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9eae3152a6a640e9ac6fcac8fccd537b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eae3152a6a640e9ac6fcac8fccd537b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34251 } }
I20260926 09:31:21.500151 20850 leader_election.cc:302] T 00000000000000000000000000000000 P 9eae3152a6a640e9ac6fcac8fccd537b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9eae3152a6a640e9ac6fcac8fccd537b; no voters:
I20260926 09:31:21.500363 20850 leader_election.cc:288] T 00000000000000000000000000000000 P 9eae3152a6a640e9ac6fcac8fccd537b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:31:21.500391 20855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9eae3152a6a640e9ac6fcac8fccd537b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:21.500519 20855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9eae3152a6a640e9ac6fcac8fccd537b [term 1 LEADER]: Becoming Leader. State: Replica: 9eae3152a6a640e9ac6fcac8fccd537b, State: Running, Role: LEADER
I20260926 09:31:21.500582 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:31:21.500630 20850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9eae3152a6a640e9ac6fcac8fccd537b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:21.500602 20855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9eae3152a6a640e9ac6fcac8fccd537b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eae3152a6a640e9ac6fcac8fccd537b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34251 } }
I20260926 09:31:21.500990 20856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eae3152a6a640e9ac6fcac8fccd537b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9eae3152a6a640e9ac6fcac8fccd537b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eae3152a6a640e9ac6fcac8fccd537b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34251 } } }
I20260926 09:31:21.501006 20857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eae3152a6a640e9ac6fcac8fccd537b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9eae3152a6a640e9ac6fcac8fccd537b. Latest consensus state: current_term: 1 leader_uuid: "9eae3152a6a640e9ac6fcac8fccd537b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eae3152a6a640e9ac6fcac8fccd537b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34251 } } }
I20260926 09:31:21.501092 20856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eae3152a6a640e9ac6fcac8fccd537b [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:21.501119 20857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eae3152a6a640e9ac6fcac8fccd537b [sys.catalog]: This master's current role is: LEADER
W20260926 09:31:21.502804 20876 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9eae3152a6a640e9ac6fcac8fccd537b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:31:21.502880 20876 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:31:21.502943 20860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:21.503124 20860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:21.503616 20854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:21.502149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58521 (local address 127.0.0.1:34251)
0926 09:31:21.502191 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:21.502193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:21.502400 (+ 207us) server_negotiation.cc:407] Connection header received
0926 09:31:21.502582 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:21.502583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:21.502589 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:21.502627 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:21.503345 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:21.503346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:21.503347 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:21.503352 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:21.503366 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:21.503370 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:21.503422 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:21.503544 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 09:31:21.503558 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:31:21.504679 20860 catalog_manager.cc:1347] Generated new cluster ID: 2d7d61faa69643fb800fe1fa8b8ce998
I20260926 09:31:21.504766 20860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:31:21.515154 20860 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:21.515337 20860 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:21.518738 20860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9eae3152a6a640e9ac6fcac8fccd537b: Generated new TSK 0
I20260926 09:31:21.518913 20860 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:31:21.522702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 20782
2026-09-26T09:31:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:31:21 dist-test-slave-dfzm krb5kdc[20881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:31:21 dist-test-slave-dfzm krb5kdc[20881](info): set up 2 sockets
Sep 26 09:31:21 dist-test-slave-dfzm krb5kdc[20881](info): commencing operation
krb5kdc: starting...
W20260926 09:31:22.553771 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:31:22 dist-test-slave-dfzm krb5kdc[20881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:31:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:31:22.668817 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36359
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:22.748708 20894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:22.748955 20894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:22.749004 20894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:22.750213 20894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:22.750284 20894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:22.750309 20894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:22.750344 20894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:22.750371 20894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:22.752583 20894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:22.752852 20894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:22.753131 20894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:22.758467 20899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:22.758469 20902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:22.758736 20894 server_base.cc:1034] running on GCE node
W20260926 09:31:22.758505 20900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:22.759090 20894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:22.759449 20894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:22.760627 20894 hybrid_clock.cc:648] HybridClock initialized: now 1790415082760606 us; error 38 us; skew 500 ppm
Sep 26 09:31:22 dist-test-slave-dfzm krb5kdc[20881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:22.763294 20894 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:31:22.763769 20894 webserver.cc:460] Webserver started at http://127.0.0.1:38601/ using document root <none> and password file <none>
I20260926 09:31:22.763974 20894 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:22.764016 20894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:22.764108 20894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:22.765144 20894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "041aa4fc3ba14d9f928454c9dbf36d61"
format_stamp: "Formatted at 2026-09-26 09:31:22 on dist-test-slave-dfzm"
server_key: "30279f1f48f3267a18af012ed9cab9f3"
server_key_iv: "67359930d657a055fe1ed079ca010379"
server_key_version: "encryptionkey@0"
I20260926 09:31:22.765448 20894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "041aa4fc3ba14d9f928454c9dbf36d61"
format_stamp: "Formatted at 2026-09-26 09:31:22 on dist-test-slave-dfzm"
server_key: "30279f1f48f3267a18af012ed9cab9f3"
server_key_iv: "67359930d657a055fe1ed079ca010379"
server_key_version: "encryptionkey@0"
I20260926 09:31:22.766816 20894 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:22.767552 20909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:22.767733 20894 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:31:22.767825 20894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "041aa4fc3ba14d9f928454c9dbf36d61"
format_stamp: "Formatted at 2026-09-26 09:31:22 on dist-test-slave-dfzm"
server_key: "30279f1f48f3267a18af012ed9cab9f3"
server_key_iv: "67359930d657a055fe1ed079ca010379"
server_key_version: "encryptionkey@0"
I20260926 09:31:22.767884 20894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:22.787292 20894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:22.789812 20894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:22.790035 20894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:22.793956 20894 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36359
I20260926 09:31:22.794023 20961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36359 every 8 connection(s)
I20260926 09:31:22.794394 20894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:22.795293 20962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:22.797572 20962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 041aa4fc3ba14d9f928454c9dbf36d61: Bootstrap starting.
I20260926 09:31:22.798233 20962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 041aa4fc3ba14d9f928454c9dbf36d61: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:22.798530 20962 log.cc:824] T 00000000000000000000000000000000 P 041aa4fc3ba14d9f928454c9dbf36d61: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:22.799312 20962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 041aa4fc3ba14d9f928454c9dbf36d61: No bootstrap required, opened a new log
I20260926 09:31:22.800905 20962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 041aa4fc3ba14d9f928454c9dbf36d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041aa4fc3ba14d9f928454c9dbf36d61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36359 } }
I20260926 09:31:22.801048 20962 raft_consensus.cc:374] T 00000000000000000000000000000000 P 041aa4fc3ba14d9f928454c9dbf36d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:22.801071 20962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 041aa4fc3ba14d9f928454c9dbf36d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 041aa4fc3ba14d9f928454c9dbf36d61, State: Initialized, Role: FOLLOWER
I20260926 09:31:22.801251 20962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 041aa4fc3ba14d9f928454c9dbf36d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041aa4fc3ba14d9f928454c9dbf36d61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36359 } }
I20260926 09:31:22.801316 20962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 041aa4fc3ba14d9f928454c9dbf36d61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:31:22.801358 20962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 041aa4fc3ba14d9f928454c9dbf36d61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:31:22.801407 20962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 041aa4fc3ba14d9f928454c9dbf36d61 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:22.802028 20962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 041aa4fc3ba14d9f928454c9dbf36d61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041aa4fc3ba14d9f928454c9dbf36d61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36359 } }
I20260926 09:31:22.802124 20962 leader_election.cc:302] T 00000000000000000000000000000000 P 041aa4fc3ba14d9f928454c9dbf36d61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 041aa4fc3ba14d9f928454c9dbf36d61; no voters:
I20260926 09:31:22.802306 20962 leader_election.cc:288] T 00000000000000000000000000000000 P 041aa4fc3ba14d9f928454c9dbf36d61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:31:22.802349 20965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 041aa4fc3ba14d9f928454c9dbf36d61 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:22.802577 20962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 041aa4fc3ba14d9f928454c9dbf36d61 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:22.802611 20965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 041aa4fc3ba14d9f928454c9dbf36d61 [term 1 LEADER]: Becoming Leader. State: Replica: 041aa4fc3ba14d9f928454c9dbf36d61, State: Running, Role: LEADER
I20260926 09:31:22.802721 20965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 041aa4fc3ba14d9f928454c9dbf36d61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041aa4fc3ba14d9f928454c9dbf36d61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36359 } }
I20260926 09:31:22.803371 20966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 041aa4fc3ba14d9f928454c9dbf36d61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "041aa4fc3ba14d9f928454c9dbf36d61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041aa4fc3ba14d9f928454c9dbf36d61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36359 } } }
I20260926 09:31:22.803460 20966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 041aa4fc3ba14d9f928454c9dbf36d61 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:22.803383 20967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 041aa4fc3ba14d9f928454c9dbf36d61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 041aa4fc3ba14d9f928454c9dbf36d61. Latest consensus state: current_term: 1 leader_uuid: "041aa4fc3ba14d9f928454c9dbf36d61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041aa4fc3ba14d9f928454c9dbf36d61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36359 } } }
I20260926 09:31:22.803507 20967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 041aa4fc3ba14d9f928454c9dbf36d61 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:22.803655 20974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:22.804047 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 20894
I20260926 09:31:22.804188 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:22.804185 20974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:22.804366 392 external_mini_cluster.cc:1428] Setting key 1a0db53562d90c5032852b04f3e093d9
I20260926 09:31:22.805895 20974 catalog_manager.cc:1347] Generated new cluster ID: 180d47194ff34448949c52efd171dd09
I20260926 09:31:22.805961 20974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:31:22 dist-test-slave-dfzm krb5kdc[20881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 09:31:22.814793 20984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:22.804886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40739 (local address 127.0.0.1:36359)
0926 09:31:22.805255 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:22.805266 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:22.805487 (+ 221us) server_negotiation.cc:407] Connection header received
0926 09:31:22.805691 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:22.805701 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:22.805763 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:22.805912 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:22.806663 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:22.807570 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:22.808373 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:22.808820 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:22.808955 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:22.811385 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:22.811389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:22.811390 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:22.811402 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:22.813114 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:22.813500 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:22.813504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:22.813504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:22.813666 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:22.813874 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:22.813875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:22.813875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:22.814145 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:22.814313 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:22.814427 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 09:31:22.814571 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":107,"threads_started":1}
I20260926 09:31:22.815776 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:31:22.823719 20984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:22.817112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47845 (local address 127.0.0.1:36359)
0926 09:31:22.817158 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:22.817160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:22.817758 (+ 598us) server_negotiation.cc:407] Connection header received
0926 09:31:22.817831 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:22.817832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:22.817843 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:22.817897 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:22.818428 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:22.818936 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:22.819676 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:22.819901 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:22.819986 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:22.821162 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:22.821163 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:22.821163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:22.821170 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:22.822601 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:22.822959 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:22.822960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:22.822960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:22.823015 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:22.823215 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:22.823216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:22.823217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:22.823325 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:22.823429 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:22.823525 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 09:31:22.823605 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:31:22.824712 20974 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:22.825199 20974 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:22.829629 20974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 041aa4fc3ba14d9f928454c9dbf36d61: Generated new TSK 0
I20260926 09:31:22.829830 20974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:31:22.837183 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 20894
2026-09-26T09:31:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:31:22 dist-test-slave-dfzm krb5kdc[20993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:31:22 dist-test-slave-dfzm krb5kdc[20993](info): set up 2 sockets
Sep 26 09:31:22 dist-test-slave-dfzm krb5kdc[20993](info): commencing operation
krb5kdc: starting...
W20260926 09:31:23.901571 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:31:23 dist-test-slave-dfzm krb5kdc[20993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:31:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:31:24.024396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45089
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:24.105818 21006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:24.106086 21006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:24.106128 21006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:24.107343 21006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:24.107429 21006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:24.107456 21006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:24.107488 21006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:24.107512 21006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:24.109095 21006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:24.109400 21006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:24.109692 21006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:24.114687 21012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:24.114729 21011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:24.114717 21014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:24.114929 21006 server_base.cc:1034] running on GCE node
I20260926 09:31:24.115149 21006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:24.115394 21006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:24.116536 21006 hybrid_clock.cc:648] HybridClock initialized: now 1790415084116524 us; error 29 us; skew 500 ppm
Sep 26 09:31:24 dist-test-slave-dfzm krb5kdc[20993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:24.119095 21006 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:31:24.119475 21006 webserver.cc:460] Webserver started at http://127.0.0.1:34957/ using document root <none> and password file <none>
I20260926 09:31:24.119664 21006 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:24.119711 21006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:24.119809 21006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:24.120819 21006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "156644f25f18449095fddfb6d0c0f9be"
format_stamp: "Formatted at 2026-09-26 09:31:24 on dist-test-slave-dfzm"
server_key: "48bd7b0aaec96877efce7bc274d26d39"
server_key_iv: "e05c9dacc0c44574a3846f52f7dc22ff"
server_key_version: "encryptionkey@0"
I20260926 09:31:24.121140 21006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "156644f25f18449095fddfb6d0c0f9be"
format_stamp: "Formatted at 2026-09-26 09:31:24 on dist-test-slave-dfzm"
server_key: "48bd7b0aaec96877efce7bc274d26d39"
server_key_iv: "e05c9dacc0c44574a3846f52f7dc22ff"
server_key_version: "encryptionkey@0"
I20260926 09:31:24.122577 21006 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:31:24.123373 21021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:24.123592 21006 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:31:24.123678 21006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "156644f25f18449095fddfb6d0c0f9be"
format_stamp: "Formatted at 2026-09-26 09:31:24 on dist-test-slave-dfzm"
server_key: "48bd7b0aaec96877efce7bc274d26d39"
server_key_iv: "e05c9dacc0c44574a3846f52f7dc22ff"
server_key_version: "encryptionkey@0"
I20260926 09:31:24.123732 21006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:24.134913 21006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:24.136919 21006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:24.137028 21006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:24.141099 21006 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45089
I20260926 09:31:24.141179 21073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45089 every 8 connection(s)
I20260926 09:31:24.141528 21006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:24.142488 21074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:24.144704 21074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 156644f25f18449095fddfb6d0c0f9be: Bootstrap starting.
I20260926 09:31:24.145279 21074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 156644f25f18449095fddfb6d0c0f9be: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:24.145565 21074 log.cc:824] T 00000000000000000000000000000000 P 156644f25f18449095fddfb6d0c0f9be: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:24.146404 21074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 156644f25f18449095fddfb6d0c0f9be: No bootstrap required, opened a new log
I20260926 09:31:24.147996 21074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 156644f25f18449095fddfb6d0c0f9be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "156644f25f18449095fddfb6d0c0f9be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45089 } }
I20260926 09:31:24.148137 21074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 156644f25f18449095fddfb6d0c0f9be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:24.148183 21074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 156644f25f18449095fddfb6d0c0f9be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156644f25f18449095fddfb6d0c0f9be, State: Initialized, Role: FOLLOWER
I20260926 09:31:24.148346 21074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 156644f25f18449095fddfb6d0c0f9be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "156644f25f18449095fddfb6d0c0f9be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45089 } }
I20260926 09:31:24.148412 21074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 156644f25f18449095fddfb6d0c0f9be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:31:24.148437 21074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 156644f25f18449095fddfb6d0c0f9be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:31:24.148485 21074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 156644f25f18449095fddfb6d0c0f9be [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:24.149008 21074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 156644f25f18449095fddfb6d0c0f9be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "156644f25f18449095fddfb6d0c0f9be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45089 } }
I20260926 09:31:24.149091 21074 leader_election.cc:302] T 00000000000000000000000000000000 P 156644f25f18449095fddfb6d0c0f9be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 156644f25f18449095fddfb6d0c0f9be; no voters:
I20260926 09:31:24.149322 21074 leader_election.cc:288] T 00000000000000000000000000000000 P 156644f25f18449095fddfb6d0c0f9be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:31:24.149360 21077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 156644f25f18449095fddfb6d0c0f9be [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:24.149427 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 21006
I20260926 09:31:24.149485 21077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 156644f25f18449095fddfb6d0c0f9be [term 1 LEADER]: Becoming Leader. State: Replica: 156644f25f18449095fddfb6d0c0f9be, State: Running, Role: LEADER
I20260926 09:31:24.149569 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:24.149556 21077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 156644f25f18449095fddfb6d0c0f9be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "156644f25f18449095fddfb6d0c0f9be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45089 } }
I20260926 09:31:24.149617 21074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 156644f25f18449095fddfb6d0c0f9be [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:24.149784 392 external_mini_cluster.cc:1428] Setting key 6297512084e3425dc5e451e85ef84713
I20260926 09:31:24.150055 21078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 156644f25f18449095fddfb6d0c0f9be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "156644f25f18449095fddfb6d0c0f9be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "156644f25f18449095fddfb6d0c0f9be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45089 } } }
I20260926 09:31:24.150146 21078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 156644f25f18449095fddfb6d0c0f9be [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:24.150511 21079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 156644f25f18449095fddfb6d0c0f9be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 156644f25f18449095fddfb6d0c0f9be. Latest consensus state: current_term: 1 leader_uuid: "156644f25f18449095fddfb6d0c0f9be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "156644f25f18449095fddfb6d0c0f9be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45089 } } }
I20260926 09:31:24.150576 21079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 156644f25f18449095fddfb6d0c0f9be [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:24.150889 21086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:24.151532 21086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:24.152997 21086 catalog_manager.cc:1347] Generated new cluster ID: 48621472502746f1aac5e6006989e8e0
I20260926 09:31:24.153049 21086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:31:24.156343 21091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:24.150606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37269 (local address 127.0.0.1:45089)
0926 09:31:24.150945 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:24.150956 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:24.150992 (+ 36us) server_negotiation.cc:407] Connection header received
0926 09:31:24.151120 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:24.151129 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:24.151191 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:24.151355 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:24.151590 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:24.152967 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:24.154110 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:24.154753 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:24.154891 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:24.155629 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:24.155633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:24.155633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:24.155841 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:24.155879 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:24.155894 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:24.155951 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:24.156071 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 09:31:24.156138 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":66,"threads_started":1}
I20260926 09:31:24.157246 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:31:24.162472 21091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:24.158863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58637 (local address 127.0.0.1:45089)
0926 09:31:24.158896 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:24.158898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:24.159049 (+ 151us) server_negotiation.cc:407] Connection header received
0926 09:31:24.159112 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:24.159112 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:24.159122 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:24.159175 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:24.159307 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:24.160038 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:24.161084 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:24.161375 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:24.161438 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:24.162204 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:24.162205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:24.162206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:24.162210 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:24.162224 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:24.162228 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:24.162278 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:24.162362 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 09:31:24.162389 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 09:31:24.173880 21086 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:24.174461 21086 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:24.179690 21086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 156644f25f18449095fddfb6d0c0f9be: Generated new TSK 0
I20260926 09:31:24.179900 21086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:31:24.185139 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 21006
2026-09-26T09:31:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1347 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:31:24 dist-test-slave-dfzm krb5kdc[21105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:31:24 dist-test-slave-dfzm krb5kdc[21105](info): set up 2 sockets
Sep 26 09:31:24 dist-test-slave-dfzm krb5kdc[21105](info): commencing operation
krb5kdc: starting...
W20260926 09:31:25.226140 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:31:25 dist-test-slave-dfzm krb5kdc[21105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:31:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:31:25.345243 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45967
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:25.426550 21118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:25.426819 21118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:25.426862 21118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:25.428089 21118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:25.428170 21118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:25.428193 21118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:25.428215 21118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:25.428244 21118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:25.429760 21118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:25.430051 21118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:25.430284 21118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:25.435236 21126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:25.435262 21124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:25.435246 21123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:25.435518 21118 server_base.cc:1034] running on GCE node
I20260926 09:31:25.435842 21118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:25.436117 21118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:25.437264 21118 hybrid_clock.cc:648] HybridClock initialized: now 1790415085437246 us; error 41 us; skew 500 ppm
Sep 26 09:31:25 dist-test-slave-dfzm krb5kdc[21105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:25.444134 21118 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:31:25.444694 21118 webserver.cc:460] Webserver started at http://127.0.0.1:46805/ using document root <none> and password file <none>
I20260926 09:31:25.444941 21118 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:25.445016 21118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:25.445195 21118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:25.446367 21118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "f2d8a0bb179847618cb3ebe4e5871c10"
format_stamp: "Formatted at 2026-09-26 09:31:25 on dist-test-slave-dfzm"
server_key: "3f3aa22a48f7be3996d8783a20c619f5"
server_key_iv: "df7d7e7b74eaa759064db8c1cc58763f"
server_key_version: "encryptionkey@0"
I20260926 09:31:25.446748 21118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "f2d8a0bb179847618cb3ebe4e5871c10"
format_stamp: "Formatted at 2026-09-26 09:31:25 on dist-test-slave-dfzm"
server_key: "3f3aa22a48f7be3996d8783a20c619f5"
server_key_iv: "df7d7e7b74eaa759064db8c1cc58763f"
server_key_version: "encryptionkey@0"
I20260926 09:31:25.448220 21118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:31:25.449064 21133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:25.449316 21118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:25.449432 21118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "f2d8a0bb179847618cb3ebe4e5871c10"
format_stamp: "Formatted at 2026-09-26 09:31:25 on dist-test-slave-dfzm"
server_key: "3f3aa22a48f7be3996d8783a20c619f5"
server_key_iv: "df7d7e7b74eaa759064db8c1cc58763f"
server_key_version: "encryptionkey@0"
I20260926 09:31:25.449512 21118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:25.452940 21118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:25.455386 21118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:25.455550 21118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:25.459923 21118 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45967
I20260926 09:31:25.459952 21185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45967 every 8 connection(s)
I20260926 09:31:25.460312 21118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:25.461333 21186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:25.463711 21186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2d8a0bb179847618cb3ebe4e5871c10: Bootstrap starting.
I20260926 09:31:25.464293 21186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2d8a0bb179847618cb3ebe4e5871c10: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:25.464632 21186 log.cc:824] T 00000000000000000000000000000000 P f2d8a0bb179847618cb3ebe4e5871c10: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:25.465364 21186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2d8a0bb179847618cb3ebe4e5871c10: No bootstrap required, opened a new log
I20260926 09:31:25.466997 21186 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2d8a0bb179847618cb3ebe4e5871c10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2d8a0bb179847618cb3ebe4e5871c10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45967 } }
I20260926 09:31:25.467166 21186 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2d8a0bb179847618cb3ebe4e5871c10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:25.467216 21186 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2d8a0bb179847618cb3ebe4e5871c10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2d8a0bb179847618cb3ebe4e5871c10, State: Initialized, Role: FOLLOWER
I20260926 09:31:25.467374 21186 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2d8a0bb179847618cb3ebe4e5871c10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2d8a0bb179847618cb3ebe4e5871c10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45967 } }
I20260926 09:31:25.467456 21186 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2d8a0bb179847618cb3ebe4e5871c10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:31:25.467505 21186 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2d8a0bb179847618cb3ebe4e5871c10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:31:25.467553 21186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2d8a0bb179847618cb3ebe4e5871c10 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:25.468097 21186 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2d8a0bb179847618cb3ebe4e5871c10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2d8a0bb179847618cb3ebe4e5871c10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45967 } }
I20260926 09:31:25.468220 21186 leader_election.cc:302] T 00000000000000000000000000000000 P f2d8a0bb179847618cb3ebe4e5871c10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2d8a0bb179847618cb3ebe4e5871c10; no voters:
I20260926 09:31:25.468406 21186 leader_election.cc:288] T 00000000000000000000000000000000 P f2d8a0bb179847618cb3ebe4e5871c10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:31:25.468442 21189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2d8a0bb179847618cb3ebe4e5871c10 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:25.468622 21189 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2d8a0bb179847618cb3ebe4e5871c10 [term 1 LEADER]: Becoming Leader. State: Replica: f2d8a0bb179847618cb3ebe4e5871c10, State: Running, Role: LEADER
I20260926 09:31:25.468674 21186 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2d8a0bb179847618cb3ebe4e5871c10 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:25.468801 21189 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2d8a0bb179847618cb3ebe4e5871c10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2d8a0bb179847618cb3ebe4e5871c10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45967 } }
I20260926 09:31:25.469204 21190 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2d8a0bb179847618cb3ebe4e5871c10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2d8a0bb179847618cb3ebe4e5871c10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2d8a0bb179847618cb3ebe4e5871c10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45967 } } }
I20260926 09:31:25.469364 21190 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2d8a0bb179847618cb3ebe4e5871c10 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:25.469213 21191 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2d8a0bb179847618cb3ebe4e5871c10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2d8a0bb179847618cb3ebe4e5871c10. Latest consensus state: current_term: 1 leader_uuid: "f2d8a0bb179847618cb3ebe4e5871c10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2d8a0bb179847618cb3ebe4e5871c10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45967 } } }
I20260926 09:31:25.469628 21191 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2d8a0bb179847618cb3ebe4e5871c10 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:25.469801 21193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:25.470167 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 21118
I20260926 09:31:25.470291 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:25.470397 21193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:25.470470 392 external_mini_cluster.cc:1428] Setting key 1510880062dd9413bcf252100aec33df
I20260926 09:31:25.472097 21193 catalog_manager.cc:1347] Generated new cluster ID: a1aa4795e8da499cb96b2692ef95dcb6
I20260926 09:31:25.472149 21193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:31:25.473217 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:25.471095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33737 (local address 127.0.0.1:45967)
0926 09:31:25.471275 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:25.471284 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:25.471314 (+ 30us) server_negotiation.cc:407] Connection header received
0926 09:31:25.471841 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:25.471849 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:25.471882 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:25.472293 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:25.472768 (+ 475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:25.472770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:25.472771 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:25.472774 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:25.472805 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:25.472817 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:25.472985 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:25.473013 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 09:31:25.473051 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
I20260926 09:31:25.478160 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:31:25.482443 21193 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:25.482602 21193 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:25.483497 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:25.482253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53927 (local address 127.0.0.1:45967)
0926 09:31:25.482292 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:25.482294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:25.482425 (+ 131us) server_negotiation.cc:407] Connection header received
0926 09:31:25.482524 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:25.482525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:25.482530 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:25.482559 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:25.483263 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:25.483265 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:25.483265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:25.483269 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:25.483287 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:25.483292 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:25.483356 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:25.483420 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 09:31:25.483434 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:31:25.489714 21193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2d8a0bb179847618cb3ebe4e5871c10: Generated new TSK 0
I20260926 09:31:25.489913 21193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:31:25.494735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 21118
2026-09-26T09:31:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:31:25 dist-test-slave-dfzm krb5kdc[21217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:31:25 dist-test-slave-dfzm krb5kdc[21217](info): set up 2 sockets
Sep 26 09:31:25 dist-test-slave-dfzm krb5kdc[21217](info): commencing operation
krb5kdc: starting...
W20260926 09:31:26.569775 392 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:31:26 dist-test-slave-dfzm krb5kdc[21217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:31:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:31:26.692895 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44597
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:26.776561 21230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:26.776824 21230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:26.776866 21230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:26.778162 21230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:26.778228 21230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:26.778262 21230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:26.778291 21230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:26.778318 21230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:26.779842 21230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:26.780134 21230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:26.780380 21230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:26.785449 21236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:26.785449 21238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:26.785625 21235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:26.785640 21230 server_base.cc:1034] running on GCE node
I20260926 09:31:26.785984 21230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:26.786376 21230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:26.787560 21230 hybrid_clock.cc:648] HybridClock initialized: now 1790415086787543 us; error 44 us; skew 500 ppm
Sep 26 09:31:26 dist-test-slave-dfzm krb5kdc[21217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:26.790096 21230 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:31:26.790566 21230 webserver.cc:460] Webserver started at http://127.0.0.1:46005/ using document root <none> and password file <none>
I20260926 09:31:26.790769 21230 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:26.790812 21230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:26.790911 21230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:26.791990 21230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "71c146b9d3234cb9a0e3ffbe58e914d3"
format_stamp: "Formatted at 2026-09-26 09:31:26 on dist-test-slave-dfzm"
server_key: "8d2b496695ddd5288968cc4f29c7e3d0"
server_key_iv: "c0f10ca2f88c66d641e88f8a43c0ff54"
server_key_version: "encryptionkey@0"
I20260926 09:31:26.792307 21230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "71c146b9d3234cb9a0e3ffbe58e914d3"
format_stamp: "Formatted at 2026-09-26 09:31:26 on dist-test-slave-dfzm"
server_key: "8d2b496695ddd5288968cc4f29c7e3d0"
server_key_iv: "c0f10ca2f88c66d641e88f8a43c0ff54"
server_key_version: "encryptionkey@0"
I20260926 09:31:26.793700 21230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 09:31:26.794518 21245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:26.794674 21230 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:31:26.794771 21230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "71c146b9d3234cb9a0e3ffbe58e914d3"
format_stamp: "Formatted at 2026-09-26 09:31:26 on dist-test-slave-dfzm"
server_key: "8d2b496695ddd5288968cc4f29c7e3d0"
server_key_iv: "c0f10ca2f88c66d641e88f8a43c0ff54"
server_key_version: "encryptionkey@0"
I20260926 09:31:26.794828 21230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:26.815128 21230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:26.817425 21230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:26.817546 21230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:26.821415 21230 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44597
I20260926 09:31:26.821470 21297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44597 every 8 connection(s)
I20260926 09:31:26.821849 21230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:26.822837 21298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:26.825064 21298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71c146b9d3234cb9a0e3ffbe58e914d3: Bootstrap starting.
I20260926 09:31:26.825699 21298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71c146b9d3234cb9a0e3ffbe58e914d3: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:26.825937 21298 log.cc:824] T 00000000000000000000000000000000 P 71c146b9d3234cb9a0e3ffbe58e914d3: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:26.826630 21298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71c146b9d3234cb9a0e3ffbe58e914d3: No bootstrap required, opened a new log
I20260926 09:31:26.828054 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 21230
I20260926 09:31:26.828188 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:26.828397 392 external_mini_cluster.cc:1428] Setting key a701634cbff7ff02a342e66503edc9fa
I20260926 09:31:26.828554 21298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71c146b9d3234cb9a0e3ffbe58e914d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c146b9d3234cb9a0e3ffbe58e914d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44597 } }
I20260926 09:31:26.828707 21298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71c146b9d3234cb9a0e3ffbe58e914d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:26.828733 21298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71c146b9d3234cb9a0e3ffbe58e914d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c146b9d3234cb9a0e3ffbe58e914d3, State: Initialized, Role: FOLLOWER
I20260926 09:31:26.828931 21298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71c146b9d3234cb9a0e3ffbe58e914d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c146b9d3234cb9a0e3ffbe58e914d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44597 } }
I20260926 09:31:26.829061 21298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71c146b9d3234cb9a0e3ffbe58e914d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:31:26.829107 21298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71c146b9d3234cb9a0e3ffbe58e914d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:31:26.829173 21298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71c146b9d3234cb9a0e3ffbe58e914d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:26.830155 21298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71c146b9d3234cb9a0e3ffbe58e914d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c146b9d3234cb9a0e3ffbe58e914d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44597 } }
I20260926 09:31:26.830278 21298 leader_election.cc:302] T 00000000000000000000000000000000 P 71c146b9d3234cb9a0e3ffbe58e914d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71c146b9d3234cb9a0e3ffbe58e914d3; no voters:
I20260926 09:31:26.830488 21298 leader_election.cc:288] T 00000000000000000000000000000000 P 71c146b9d3234cb9a0e3ffbe58e914d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:31:26.830518 21303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71c146b9d3234cb9a0e3ffbe58e914d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:26.830632 21303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71c146b9d3234cb9a0e3ffbe58e914d3 [term 1 LEADER]: Becoming Leader. State: Replica: 71c146b9d3234cb9a0e3ffbe58e914d3, State: Running, Role: LEADER
I20260926 09:31:26.830765 21298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71c146b9d3234cb9a0e3ffbe58e914d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:26.830767 21303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71c146b9d3234cb9a0e3ffbe58e914d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c146b9d3234cb9a0e3ffbe58e914d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44597 } }
I20260926 09:31:26.831212 21305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c146b9d3234cb9a0e3ffbe58e914d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71c146b9d3234cb9a0e3ffbe58e914d3. Latest consensus state: current_term: 1 leader_uuid: "71c146b9d3234cb9a0e3ffbe58e914d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c146b9d3234cb9a0e3ffbe58e914d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44597 } } }
I20260926 09:31:26.831338 21305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c146b9d3234cb9a0e3ffbe58e914d3 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:26.831519 21304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c146b9d3234cb9a0e3ffbe58e914d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71c146b9d3234cb9a0e3ffbe58e914d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c146b9d3234cb9a0e3ffbe58e914d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44597 } } }
I20260926 09:31:26.831606 21304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c146b9d3234cb9a0e3ffbe58e914d3 [sys.catalog]: This master's current role is: LEADER
W20260926 09:31:26.832631 21319 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71c146b9d3234cb9a0e3ffbe58e914d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:31:26.832697 21319 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:31:26.832794 21312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:26.832968 21312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:26.834631 21312 catalog_manager.cc:1347] Generated new cluster ID: be17fb87373541bd8a88ce80fef733d4
I20260926 09:31:26.834688 21312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:31:26 dist-test-slave-dfzm krb5kdc[21217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 09:31:26.839745 21302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:26.829103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:40039 (local address 127.0.0.1:44597)
0926 09:31:26.829309 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:26.829320 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:26.829353 (+ 33us) server_negotiation.cc:407] Connection header received
0926 09:31:26.829463 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:26.829469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:26.829527 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:26.829702 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:26.830413 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:26.831810 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:26.832946 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:26.833554 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:26.833725 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:26.836211 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:26.836216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:26.836216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:26.836231 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:26.838037 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:26.838452 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:26.838455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:26.838458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:26.838561 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:26.838805 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:26.838806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:26.838806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:26.839057 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:26.839232 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:26.839359 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 09:31:26.839571 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":96,"threads_started":1}
I20260926 09:31:26.840758 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:31:26.851936 21302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:26.842271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:44319 (local address 127.0.0.1:44597)
0926 09:31:26.842312 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:26.842314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:26.845772 (+ 3458us) server_negotiation.cc:407] Connection header received
0926 09:31:26.845884 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:26.845885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:26.845898 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:26.845943 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:26.846513 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:26.847040 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:26.847762 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:26.847981 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:26.848054 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:26.849280 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:26.849281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:26.849282 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:26.849289 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:26.850757 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:26.851133 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:26.851134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:26.851135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:26.851195 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:26.851417 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:26.851418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:26.851418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:26.851531 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:26.851653 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:26.851770 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 09:31:26.851834 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:31:26.854606 21312 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:26.855139 21312 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:26.860589 21312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71c146b9d3234cb9a0e3ffbe58e914d3: Generated new TSK 0
I20260926 09:31:26.860805 21312 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:31:26.865509 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 21230
W20260926 09:31:26.869956 21215 connection.cc:535] client connection to 127.0.0.1:44597 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44597): Connection reset by peer (error 104)
2026-09-26T09:31:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:31:26 dist-test-slave-dfzm krb5kdc[21329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:31:26 dist-test-slave-dfzm krb5kdc[21329](info): set up 2 sockets
Sep 26 09:31:26 dist-test-slave-dfzm krb5kdc[21329](info): commencing operation
krb5kdc: starting...
W20260926 09:31:27.901917 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:31:27 dist-test-slave-dfzm krb5kdc[21329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:31:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:31:28.020788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39959
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:28.105094 21342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:28.105356 21342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:28.105398 21342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:28.106606 21342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:28.106680 21342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:28.106703 21342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:28.106726 21342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:28.106755 21342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:28.108253 21342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:28.108538 21342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:28.108769 21342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:28.113656 21348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:28.113703 21347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:28.113883 21342 server_base.cc:1034] running on GCE node
W20260926 09:31:28.113722 21350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:28.114236 21342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:28.114499 21342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:28.115644 21342 hybrid_clock.cc:648] HybridClock initialized: now 1790415088115624 us; error 34 us; skew 500 ppm
Sep 26 09:31:28 dist-test-slave-dfzm krb5kdc[21329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:28.118074 21342 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:31:28.118506 21342 webserver.cc:460] Webserver started at http://127.0.0.1:34079/ using document root <none> and password file <none>
I20260926 09:31:28.118736 21342 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:28.118806 21342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:28.118939 21342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:28.120005 21342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "f43e3b51ac004cf9af5adf6f2eea340f"
format_stamp: "Formatted at 2026-09-26 09:31:28 on dist-test-slave-dfzm"
server_key: "b22ea114804b3c1eabcdbc3bf2b45bc3"
server_key_iv: "34b29cac8e5097e034203c2522caaf64"
server_key_version: "encryptionkey@0"
I20260926 09:31:28.120375 21342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "f43e3b51ac004cf9af5adf6f2eea340f"
format_stamp: "Formatted at 2026-09-26 09:31:28 on dist-test-slave-dfzm"
server_key: "b22ea114804b3c1eabcdbc3bf2b45bc3"
server_key_iv: "34b29cac8e5097e034203c2522caaf64"
server_key_version: "encryptionkey@0"
I20260926 09:31:28.121836 21342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:31:28.122669 21357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:28.122869 21342 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:28.122969 21342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "f43e3b51ac004cf9af5adf6f2eea340f"
format_stamp: "Formatted at 2026-09-26 09:31:28 on dist-test-slave-dfzm"
server_key: "b22ea114804b3c1eabcdbc3bf2b45bc3"
server_key_iv: "34b29cac8e5097e034203c2522caaf64"
server_key_version: "encryptionkey@0"
I20260926 09:31:28.123051 21342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:28.140094 21342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:28.142119 21342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:28.142285 21342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:28.146389 21342 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39959
I20260926 09:31:28.146425 21409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39959 every 8 connection(s)
I20260926 09:31:28.146832 21342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:28.147722 21410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:28.150156 21410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f43e3b51ac004cf9af5adf6f2eea340f: Bootstrap starting.
I20260926 09:31:28.150749 21410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f43e3b51ac004cf9af5adf6f2eea340f: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:28.151039 21410 log.cc:824] T 00000000000000000000000000000000 P f43e3b51ac004cf9af5adf6f2eea340f: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:28.151741 21410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f43e3b51ac004cf9af5adf6f2eea340f: No bootstrap required, opened a new log
I20260926 09:31:28.153322 21410 raft_consensus.cc:348] T 00000000000000000000000000000000 P f43e3b51ac004cf9af5adf6f2eea340f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43e3b51ac004cf9af5adf6f2eea340f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39959 } }
I20260926 09:31:28.153461 21410 raft_consensus.cc:374] T 00000000000000000000000000000000 P f43e3b51ac004cf9af5adf6f2eea340f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:28.153484 21410 raft_consensus.cc:729] T 00000000000000000000000000000000 P f43e3b51ac004cf9af5adf6f2eea340f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f43e3b51ac004cf9af5adf6f2eea340f, State: Initialized, Role: FOLLOWER
I20260926 09:31:28.153679 21410 consensus_queue.cc:260] T 00000000000000000000000000000000 P f43e3b51ac004cf9af5adf6f2eea340f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43e3b51ac004cf9af5adf6f2eea340f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39959 } }
I20260926 09:31:28.153746 21410 raft_consensus.cc:388] T 00000000000000000000000000000000 P f43e3b51ac004cf9af5adf6f2eea340f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:31:28.153769 21410 raft_consensus.cc:482] T 00000000000000000000000000000000 P f43e3b51ac004cf9af5adf6f2eea340f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:31:28.153831 21410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f43e3b51ac004cf9af5adf6f2eea340f [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:28.154397 21410 raft_consensus.cc:504] T 00000000000000000000000000000000 P f43e3b51ac004cf9af5adf6f2eea340f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43e3b51ac004cf9af5adf6f2eea340f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39959 } }
I20260926 09:31:28.154489 21410 leader_election.cc:302] T 00000000000000000000000000000000 P f43e3b51ac004cf9af5adf6f2eea340f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f43e3b51ac004cf9af5adf6f2eea340f; no voters:
I20260926 09:31:28.154690 21410 leader_election.cc:288] T 00000000000000000000000000000000 P f43e3b51ac004cf9af5adf6f2eea340f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:31:28.154834 21413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f43e3b51ac004cf9af5adf6f2eea340f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:28.154946 21410 sys_catalog.cc:564] T 00000000000000000000000000000000 P f43e3b51ac004cf9af5adf6f2eea340f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:28.155074 21413 raft_consensus.cc:686] T 00000000000000000000000000000000 P f43e3b51ac004cf9af5adf6f2eea340f [term 1 LEADER]: Becoming Leader. State: Replica: f43e3b51ac004cf9af5adf6f2eea340f, State: Running, Role: LEADER
I20260926 09:31:28.155170 21413 consensus_queue.cc:237] T 00000000000000000000000000000000 P f43e3b51ac004cf9af5adf6f2eea340f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43e3b51ac004cf9af5adf6f2eea340f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39959 } }
I20260926 09:31:28.155748 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 21342
I20260926 09:31:28.155874 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:28.156081 392 external_mini_cluster.cc:1428] Setting key 98048b3eaa61163481e79611d89e71e9
I20260926 09:31:28.156585 21414 sys_catalog.cc:455] T 00000000000000000000000000000000 P f43e3b51ac004cf9af5adf6f2eea340f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f43e3b51ac004cf9af5adf6f2eea340f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43e3b51ac004cf9af5adf6f2eea340f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39959 } } }
W20260926 09:31:28.156895 21427 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f43e3b51ac004cf9af5adf6f2eea340f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 09:31:28.156939 21414 sys_catalog.cc:458] T 00000000000000000000000000000000 P f43e3b51ac004cf9af5adf6f2eea340f [sys.catalog]: This master's current role is: LEADER
W20260926 09:31:28.156945 21427 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:31:28.157131 21415 sys_catalog.cc:455] T 00000000000000000000000000000000 P f43e3b51ac004cf9af5adf6f2eea340f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f43e3b51ac004cf9af5adf6f2eea340f. Latest consensus state: current_term: 1 leader_uuid: "f43e3b51ac004cf9af5adf6f2eea340f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43e3b51ac004cf9af5adf6f2eea340f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39959 } } }
I20260926 09:31:28.157215 21415 sys_catalog.cc:458] T 00000000000000000000000000000000 P f43e3b51ac004cf9af5adf6f2eea340f [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:28.157337 21432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:28.157604 21432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:28.158968 21432 catalog_manager.cc:1347] Generated new cluster ID: 0a0cf9b5f52d41f6b3633b82d01f3dc2
I20260926 09:31:28.159020 21432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:31:28.162940 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:28.156782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:54627 (local address 127.0.0.1:39959)
0926 09:31:28.156969 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:28.156979 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:28.157010 (+ 31us) server_negotiation.cc:407] Connection header received
0926 09:31:28.157136 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:28.157144 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:28.157208 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:28.157350 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:28.157621 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:28.159201 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:28.160303 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:28.160956 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:28.161072 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:28.161873 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:28.161878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:28.161878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:28.162099 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:28.162197 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:28.162251 (+ 54us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:28.162353 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:28.162658 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 09:31:28.162698 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260926 09:31:28.164072 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:31:28.169166 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:28.165632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:46149 (local address 127.0.0.1:39959)
0926 09:31:28.165671 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:28.165673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:28.165815 (+ 142us) server_negotiation.cc:407] Connection header received
0926 09:31:28.165875 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:28.165876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:28.165887 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:28.165934 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:28.166080 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:28.166799 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:28.167819 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:28.168122 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:28.168180 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:28.168907 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:28.168909 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:28.168909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:28.168914 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:28.168927 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:28.168932 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:28.168982 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:28.169076 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 09:31:28.169088 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:31:28.173305 21432 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:28.173920 21432 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:28.177562 21432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f43e3b51ac004cf9af5adf6f2eea340f: Generated new TSK 0
I20260926 09:31:28.177766 21432 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:31:28.181763 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 21342
W20260926 09:31:28.185899 21327 connection.cc:535] client connection to 127.0.0.1:39959 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39959): Connection reset by peer (error 104)
2026-09-26T09:31:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:31:28 dist-test-slave-dfzm krb5kdc[21441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:31:28 dist-test-slave-dfzm krb5kdc[21441](info): set up 2 sockets
Sep 26 09:31:28 dist-test-slave-dfzm krb5kdc[21441](info): commencing operation
krb5kdc: starting...
W20260926 09:31:29.213728 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:31:29 dist-test-slave-dfzm krb5kdc[21441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:31:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:31:29.337312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34585
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:29.418870 21454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:29.419149 21454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:29.419191 21454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:29.420413 21454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:29.420496 21454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:29.420523 21454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:29.420554 21454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:29.420579 21454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:29.422227 21454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:29.422521 21454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:29.422761 21454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:29.427824 21460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:29.427855 21459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:29.427870 21462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:29.428269 21454 server_base.cc:1034] running on GCE node
I20260926 09:31:29.428522 21454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:29.428784 21454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:29.429926 21454 hybrid_clock.cc:648] HybridClock initialized: now 1790415089429924 us; error 41 us; skew 500 ppm
Sep 26 09:31:29 dist-test-slave-dfzm krb5kdc[21441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:29.432771 21454 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:31:29.433300 21454 webserver.cc:460] Webserver started at http://127.0.0.1:36363/ using document root <none> and password file <none>
I20260926 09:31:29.433655 21454 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:29.433733 21454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:29.433926 21454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:29.435153 21454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "9b75155ca6ee493aa60a7a2acedf412d"
format_stamp: "Formatted at 2026-09-26 09:31:29 on dist-test-slave-dfzm"
server_key: "262c01aa2e51a04b254be41135e119b4"
server_key_iv: "aafe4900f2be05ea04d1b45ad2309455"
server_key_version: "encryptionkey@0"
I20260926 09:31:29.435575 21454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "9b75155ca6ee493aa60a7a2acedf412d"
format_stamp: "Formatted at 2026-09-26 09:31:29 on dist-test-slave-dfzm"
server_key: "262c01aa2e51a04b254be41135e119b4"
server_key_iv: "aafe4900f2be05ea04d1b45ad2309455"
server_key_version: "encryptionkey@0"
I20260926 09:31:29.437120 21454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:31:29.438030 21469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:29.438278 21454 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:31:29.438400 21454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "9b75155ca6ee493aa60a7a2acedf412d"
format_stamp: "Formatted at 2026-09-26 09:31:29 on dist-test-slave-dfzm"
server_key: "262c01aa2e51a04b254be41135e119b4"
server_key_iv: "aafe4900f2be05ea04d1b45ad2309455"
server_key_version: "encryptionkey@0"
I20260926 09:31:29.438493 21454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:29.442401 21454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:29.444636 21454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:29.444794 21454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:29.449055 21454 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34585
I20260926 09:31:29.449106 21521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34585 every 8 connection(s)
I20260926 09:31:29.449470 21454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:29.450556 21522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:29.452235 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 21454
I20260926 09:31:29.452394 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:29.452613 392 external_mini_cluster.cc:1428] Setting key 0c062b80047b8a610f61ce3b1fcb339e
I20260926 09:31:29.453250 21522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b75155ca6ee493aa60a7a2acedf412d: Bootstrap starting.
I20260926 09:31:29.453989 21522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b75155ca6ee493aa60a7a2acedf412d: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:29.454288 21522 log.cc:824] T 00000000000000000000000000000000 P 9b75155ca6ee493aa60a7a2acedf412d: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:29.455207 21522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b75155ca6ee493aa60a7a2acedf412d: No bootstrap required, opened a new log
I20260926 09:31:29.455397 21525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:29.453195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:36897 (local address 127.0.0.1:34585)
0926 09:31:29.453387 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:29.453403 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:29.453429 (+ 26us) server_negotiation.cc:407] Connection header received
0926 09:31:29.453932 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:29.453940 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:29.453968 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:29.454135 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:29.454879 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:29.454883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:29.454883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:29.454889 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:29.454937 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:29.454952 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:29.455051 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:29.455136 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 09:31:29.455180 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260926 09:31:29.456394 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:31:29.457772 21522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b75155ca6ee493aa60a7a2acedf412d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b75155ca6ee493aa60a7a2acedf412d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34585 } }
I20260926 09:31:29.457964 21522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b75155ca6ee493aa60a7a2acedf412d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:29.457999 21522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b75155ca6ee493aa60a7a2acedf412d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b75155ca6ee493aa60a7a2acedf412d, State: Initialized, Role: FOLLOWER
I20260926 09:31:29.458156 21522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b75155ca6ee493aa60a7a2acedf412d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b75155ca6ee493aa60a7a2acedf412d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34585 } }
I20260926 09:31:29.458259 21522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b75155ca6ee493aa60a7a2acedf412d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:31:29.458294 21522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b75155ca6ee493aa60a7a2acedf412d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:31:29.458351 21522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b75155ca6ee493aa60a7a2acedf412d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:29.459273 21522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b75155ca6ee493aa60a7a2acedf412d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b75155ca6ee493aa60a7a2acedf412d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34585 } }
I20260926 09:31:29.459374 21525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:29.458187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:59367 (local address 127.0.0.1:34585)
0926 09:31:29.458252 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:29.458254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:29.458263 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:31:29.458351 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:29.458352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:29.458357 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:29.458404 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:29.459106 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:29.459108 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:29.459108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:29.459113 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:29.459129 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:29.459133 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:29.459184 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:29.459240 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 09:31:29.459256 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 09:31:29.459438 21522 leader_election.cc:302] T 00000000000000000000000000000000 P 9b75155ca6ee493aa60a7a2acedf412d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b75155ca6ee493aa60a7a2acedf412d; no voters:
I20260926 09:31:29.459632 21522 leader_election.cc:288] T 00000000000000000000000000000000 P 9b75155ca6ee493aa60a7a2acedf412d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:31:29.459683 21532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b75155ca6ee493aa60a7a2acedf412d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:29.459824 21532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b75155ca6ee493aa60a7a2acedf412d [term 1 LEADER]: Becoming Leader. State: Replica: 9b75155ca6ee493aa60a7a2acedf412d, State: Running, Role: LEADER
I20260926 09:31:29.459893 21522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b75155ca6ee493aa60a7a2acedf412d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:29.459928 21532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b75155ca6ee493aa60a7a2acedf412d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b75155ca6ee493aa60a7a2acedf412d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34585 } }
I20260926 09:31:29.460273 21533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b75155ca6ee493aa60a7a2acedf412d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b75155ca6ee493aa60a7a2acedf412d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b75155ca6ee493aa60a7a2acedf412d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34585 } } }
I20260926 09:31:29.460295 21534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b75155ca6ee493aa60a7a2acedf412d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b75155ca6ee493aa60a7a2acedf412d. Latest consensus state: current_term: 1 leader_uuid: "9b75155ca6ee493aa60a7a2acedf412d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b75155ca6ee493aa60a7a2acedf412d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34585 } } }
I20260926 09:31:29.460352 21533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b75155ca6ee493aa60a7a2acedf412d [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:29.460382 21534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b75155ca6ee493aa60a7a2acedf412d [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:29.460968 21539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:29.461464 21539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:29.463232 21539 catalog_manager.cc:1347] Generated new cluster ID: 1281538902a045dbb7efbcbdde49f6ac
I20260926 09:31:29.463281 21539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:31:29.471984 21539 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:29.472167 21539 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:29.479697 21539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b75155ca6ee493aa60a7a2acedf412d: Generated new TSK 0
I20260926 09:31:29.479902 21539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:31:29.484872 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 21454
2026-09-26T09:31:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:31:29 dist-test-slave-dfzm krb5kdc[21553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:31:29 dist-test-slave-dfzm krb5kdc[21553](info): set up 2 sockets
Sep 26 09:31:29 dist-test-slave-dfzm krb5kdc[21553](info): commencing operation
krb5kdc: starting...
W20260926 09:31:30.566164 392 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:31:30 dist-test-slave-dfzm krb5kdc[21553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:31:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:31:30.684789 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34631
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:30.767941 21566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:30.768165 21566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:30.768190 21566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:30.769459 21566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:30.769503 21566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:30.769518 21566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:30.769531 21566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:30.769543 21566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:30.771095 21566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34631
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:30.771347 21566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:30.771616 21566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:30.776727 21571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:30.776714 21572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:30.776719 21574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:30.777002 21566 server_base.cc:1034] running on GCE node
I20260926 09:31:30.777274 21566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:30.777623 21566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:30.778817 21566 hybrid_clock.cc:648] HybridClock initialized: now 1790415090778795 us; error 38 us; skew 500 ppm
Sep 26 09:31:30 dist-test-slave-dfzm krb5kdc[21553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:30.781503 21566 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:31:30.782052 21566 webserver.cc:460] Webserver started at http://127.0.0.1:44473/ using document root <none> and password file <none>
I20260926 09:31:30.782310 21566 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:30.782387 21566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:30.782543 21566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:30.783643 21566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "2bb9e5ae5d224bc1937de531aebf7cd2"
format_stamp: "Formatted at 2026-09-26 09:31:30 on dist-test-slave-dfzm"
server_key: "74f32c2f5e5f5c795881997ecb6fcd23"
server_key_iv: "f1e1326e294946a06a89c3e996436260"
server_key_version: "encryptionkey@0"
I20260926 09:31:30.784008 21566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "2bb9e5ae5d224bc1937de531aebf7cd2"
format_stamp: "Formatted at 2026-09-26 09:31:30 on dist-test-slave-dfzm"
server_key: "74f32c2f5e5f5c795881997ecb6fcd23"
server_key_iv: "f1e1326e294946a06a89c3e996436260"
server_key_version: "encryptionkey@0"
I20260926 09:31:30.785352 21566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:31:30.786258 21581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:30.786515 21566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:31:30.786625 21566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "2bb9e5ae5d224bc1937de531aebf7cd2"
format_stamp: "Formatted at 2026-09-26 09:31:30 on dist-test-slave-dfzm"
server_key: "74f32c2f5e5f5c795881997ecb6fcd23"
server_key_iv: "f1e1326e294946a06a89c3e996436260"
server_key_version: "encryptionkey@0"
I20260926 09:31:30.786713 21566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:30.813997 21566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:30.816069 21566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:30.816249 21566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:30.820732 21566 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34631
I20260926 09:31:30.820760 21633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34631 every 8 connection(s)
I20260926 09:31:30.821126 21566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:30.822054 21634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:30.824266 21634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bb9e5ae5d224bc1937de531aebf7cd2: Bootstrap starting.
I20260926 09:31:30.824805 21634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bb9e5ae5d224bc1937de531aebf7cd2: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:30.825043 21634 log.cc:824] T 00000000000000000000000000000000 P 2bb9e5ae5d224bc1937de531aebf7cd2: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:30.825799 21634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bb9e5ae5d224bc1937de531aebf7cd2: No bootstrap required, opened a new log
I20260926 09:31:30.827366 21634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bb9e5ae5d224bc1937de531aebf7cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bb9e5ae5d224bc1937de531aebf7cd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34631 } }
I20260926 09:31:30.827508 21634 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bb9e5ae5d224bc1937de531aebf7cd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:30.827533 21634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bb9e5ae5d224bc1937de531aebf7cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bb9e5ae5d224bc1937de531aebf7cd2, State: Initialized, Role: FOLLOWER
I20260926 09:31:30.827677 21634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bb9e5ae5d224bc1937de531aebf7cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bb9e5ae5d224bc1937de531aebf7cd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34631 } }
I20260926 09:31:30.827759 21634 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bb9e5ae5d224bc1937de531aebf7cd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:31:30.827788 21634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bb9e5ae5d224bc1937de531aebf7cd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:31:30.827822 21634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bb9e5ae5d224bc1937de531aebf7cd2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:30.828330 21634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bb9e5ae5d224bc1937de531aebf7cd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bb9e5ae5d224bc1937de531aebf7cd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34631 } }
I20260926 09:31:30.828418 21634 leader_election.cc:302] T 00000000000000000000000000000000 P 2bb9e5ae5d224bc1937de531aebf7cd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bb9e5ae5d224bc1937de531aebf7cd2; no voters:
I20260926 09:31:30.828542 21634 leader_election.cc:288] T 00000000000000000000000000000000 P 2bb9e5ae5d224bc1937de531aebf7cd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:31:30.828612 21637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bb9e5ae5d224bc1937de531aebf7cd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:30.828847 21637 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bb9e5ae5d224bc1937de531aebf7cd2 [term 1 LEADER]: Becoming Leader. State: Replica: 2bb9e5ae5d224bc1937de531aebf7cd2, State: Running, Role: LEADER
I20260926 09:31:30.828890 21634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bb9e5ae5d224bc1937de531aebf7cd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:30.828969 21637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bb9e5ae5d224bc1937de531aebf7cd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bb9e5ae5d224bc1937de531aebf7cd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34631 } }
I20260926 09:31:30.829551 21638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bb9e5ae5d224bc1937de531aebf7cd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bb9e5ae5d224bc1937de531aebf7cd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bb9e5ae5d224bc1937de531aebf7cd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34631 } } }
I20260926 09:31:30.829705 21639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bb9e5ae5d224bc1937de531aebf7cd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bb9e5ae5d224bc1937de531aebf7cd2. Latest consensus state: current_term: 1 leader_uuid: "2bb9e5ae5d224bc1937de531aebf7cd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bb9e5ae5d224bc1937de531aebf7cd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34631 } } }
I20260926 09:31:30.829804 21639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bb9e5ae5d224bc1937de531aebf7cd2 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:30.829741 21638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bb9e5ae5d224bc1937de531aebf7cd2 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:30.830121 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 21566
I20260926 09:31:30.830272 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:30.830482 392 external_mini_cluster.cc:1428] Setting key 5ed906057475765372abb354e145e709
W20260926 09:31:30.830741 21653 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2bb9e5ae5d224bc1937de531aebf7cd2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:31:30.830806 21653 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:31:30.830852 21647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:30.830992 21647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:30.832679 21647 catalog_manager.cc:1347] Generated new cluster ID: 19eeda5a6f2f46599fedc4d350010208
I20260926 09:31:30.832734 21647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:31:30 dist-test-slave-dfzm krb5kdc[21553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 09:31:30.841440 21656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:30.831168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:60151 (local address 127.0.0.1:34631)
0926 09:31:30.831869 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:30.831878 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:30.831909 (+ 31us) server_negotiation.cc:407] Connection header received
0926 09:31:30.832025 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:30.832032 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:30.832091 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:30.832240 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:30.833014 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:30.834097 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:30.834893 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:30.835360 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:30.835508 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:30.837988 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:30.837993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:30.837993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:30.838005 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:30.839745 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:30.840150 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:30.840153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:30.840154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:30.840265 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:30.840509 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:30.840510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:30.840510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:30.840762 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:30.840937 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:30.841050 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 09:31:30.841259 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":61,"threads_started":1}
I20260926 09:31:30.842406 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:31:30.846670 21647 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:30.847364 21647 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:30.853078 21656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:30.843776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:56341 (local address 127.0.0.1:34631)
0926 09:31:30.843814 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:30.843816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:30.845812 (+ 1996us) server_negotiation.cc:407] Connection header received
0926 09:31:30.845888 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:30.845889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:30.845900 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:30.845954 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:30.846650 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:30.847180 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:30.848021 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:30.848361 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:30.848411 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:30.850021 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:30.850022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:30.850022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:30.850029 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:30.851658 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:30.852069 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:30.852070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:30.852070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:30.852123 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:30.852341 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:30.852341 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:30.852342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:30.852457 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:30.852540 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:30.852932 (+ 392us) server_negotiation.cc:299] Negotiation successful
0926 09:31:30.852991 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:31:30.854492 21647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bb9e5ae5d224bc1937de531aebf7cd2: Generated new TSK 0
I20260926 09:31:30.854815 21647 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:31:30.862345 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 21566
W20260926 09:31:30.866667 21551 connection.cc:535] client connection to 127.0.0.1:34631 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34631): Connection reset by peer (error 104)
2026-09-26T09:31:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:31:30 dist-test-slave-dfzm krb5kdc[21665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:31:30 dist-test-slave-dfzm krb5kdc[21665](info): set up 2 sockets
Sep 26 09:31:30 dist-test-slave-dfzm krb5kdc[21665](info): commencing operation
krb5kdc: starting...
W20260926 09:31:31.897738 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:31:31 dist-test-slave-dfzm krb5kdc[21665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:31:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:31:32.017047 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41961
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:32.098794 21678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:32.099012 21678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:32.099036 21678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:32.100234 21678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:32.100279 21678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:32.100293 21678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:32.100306 21678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:32.100359 21678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:32.101861 21678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41961
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:32.102129 21678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:32.102372 21678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:32.107180 21684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:32.107193 21683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:32.107345 21686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:32.107527 21678 server_base.cc:1034] running on GCE node
I20260926 09:31:32.107777 21678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:32.108067 21678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:32.109246 21678 hybrid_clock.cc:648] HybridClock initialized: now 1790415092109241 us; error 37 us; skew 500 ppm
Sep 26 09:31:32 dist-test-slave-dfzm krb5kdc[21665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:32.111692 21678 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:31:32.112174 21678 webserver.cc:460] Webserver started at http://127.0.0.1:39089/ using document root <none> and password file <none>
I20260926 09:31:32.112443 21678 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:32.112509 21678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:32.112624 21678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:32.113704 21678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "5dce72fad7fd4471b8bc009390266005"
format_stamp: "Formatted at 2026-09-26 09:31:32 on dist-test-slave-dfzm"
server_key: "24c9725a5aaeb6b4bf6f2032f7e13b15"
server_key_iv: "fe2fc885926df7fe0610bd89335791ce"
server_key_version: "encryptionkey@0"
I20260926 09:31:32.114012 21678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "5dce72fad7fd4471b8bc009390266005"
format_stamp: "Formatted at 2026-09-26 09:31:32 on dist-test-slave-dfzm"
server_key: "24c9725a5aaeb6b4bf6f2032f7e13b15"
server_key_iv: "fe2fc885926df7fe0610bd89335791ce"
server_key_version: "encryptionkey@0"
I20260926 09:31:32.115224 21678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:32.115882 21693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:32.116073 21678 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:31:32.116187 21678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "5dce72fad7fd4471b8bc009390266005"
format_stamp: "Formatted at 2026-09-26 09:31:32 on dist-test-slave-dfzm"
server_key: "24c9725a5aaeb6b4bf6f2032f7e13b15"
server_key_iv: "fe2fc885926df7fe0610bd89335791ce"
server_key_version: "encryptionkey@0"
I20260926 09:31:32.116262 21678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:32.132208 21678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:32.134356 21678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:32.134519 21678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:32.138911 21678 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41961
I20260926 09:31:32.138945 21745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41961 every 8 connection(s)
I20260926 09:31:32.139303 21678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:32.140266 21746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:32.142096 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 21678
I20260926 09:31:32.142225 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:32.142413 392 external_mini_cluster.cc:1428] Setting key 0ee3587070849c9e95450a18ddcb113f
I20260926 09:31:32.142984 21746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dce72fad7fd4471b8bc009390266005: Bootstrap starting.
I20260926 09:31:32.143577 21746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dce72fad7fd4471b8bc009390266005: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:32.143924 21746 log.cc:824] T 00000000000000000000000000000000 P 5dce72fad7fd4471b8bc009390266005: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:32.144649 21746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dce72fad7fd4471b8bc009390266005: No bootstrap required, opened a new log
I20260926 09:31:32.146844 21746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dce72fad7fd4471b8bc009390266005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dce72fad7fd4471b8bc009390266005" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41961 } }
I20260926 09:31:32.147014 21746 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5dce72fad7fd4471b8bc009390266005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:32.147059 21746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dce72fad7fd4471b8bc009390266005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dce72fad7fd4471b8bc009390266005, State: Initialized, Role: FOLLOWER
I20260926 09:31:32.147257 21746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dce72fad7fd4471b8bc009390266005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dce72fad7fd4471b8bc009390266005" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41961 } }
I20260926 09:31:32.147327 21746 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5dce72fad7fd4471b8bc009390266005 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:31:32.147347 21746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dce72fad7fd4471b8bc009390266005 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:31:32.147423 21746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dce72fad7fd4471b8bc009390266005 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:32.148149 21746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dce72fad7fd4471b8bc009390266005 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dce72fad7fd4471b8bc009390266005" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41961 } }
I20260926 09:31:32.148262 21746 leader_election.cc:302] T 00000000000000000000000000000000 P 5dce72fad7fd4471b8bc009390266005 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5dce72fad7fd4471b8bc009390266005; no voters:
I20260926 09:31:32.148447 21751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dce72fad7fd4471b8bc009390266005 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:32.148422 21746 leader_election.cc:288] T 00000000000000000000000000000000 P 5dce72fad7fd4471b8bc009390266005 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:31:32.148612 21751 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dce72fad7fd4471b8bc009390266005 [term 1 LEADER]: Becoming Leader. State: Replica: 5dce72fad7fd4471b8bc009390266005, State: Running, Role: LEADER
I20260926 09:31:32.148672 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:32.142991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:37175 (local address 127.0.0.1:41961)
0926 09:31:32.143183 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:32.143199 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:32.143224 (+ 25us) server_negotiation.cc:407] Connection header received
0926 09:31:32.143655 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:32.143663 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:32.143730 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:32.143843 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:32.144115 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:32.145493 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:32.146338 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:32.146981 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:32.147094 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:32.147869 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:32.147873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:32.147873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:32.148084 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:32.148125 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:32.148141 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:32.148233 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:32.148374 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 09:31:32.148406 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":92,"threads_started":1}
I20260926 09:31:32.148739 21751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dce72fad7fd4471b8bc009390266005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dce72fad7fd4471b8bc009390266005" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41961 } }
I20260926 09:31:32.148737 21746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dce72fad7fd4471b8bc009390266005 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:32.149387 21752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dce72fad7fd4471b8bc009390266005 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dce72fad7fd4471b8bc009390266005. Latest consensus state: current_term: 1 leader_uuid: "5dce72fad7fd4471b8bc009390266005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dce72fad7fd4471b8bc009390266005" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41961 } } }
I20260926 09:31:32.149693 21752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dce72fad7fd4471b8bc009390266005 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:32.149890 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:31:32.149574 21753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dce72fad7fd4471b8bc009390266005 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5dce72fad7fd4471b8bc009390266005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dce72fad7fd4471b8bc009390266005" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41961 } } }
I20260926 09:31:32.149984 21753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dce72fad7fd4471b8bc009390266005 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:32.150086 21760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:32.150566 21760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:32.152045 21760 catalog_manager.cc:1347] Generated new cluster ID: fc65d3e240b644a9b2d602fe6998e5fb
I20260926 09:31:32.152110 21760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:31:32.161324 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:32.153896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:60565 (local address 127.0.0.1:41961)
0926 09:31:32.153940 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:32.153942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:32.157799 (+ 3857us) server_negotiation.cc:407] Connection header received
0926 09:31:32.157868 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:32.157869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:32.157880 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:32.157933 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:32.158098 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:32.158845 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:32.159920 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:32.160250 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:32.160316 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:32.161057 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:32.161058 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:32.161059 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:32.161063 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:32.161080 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:32.161084 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:32.161140 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:32.161223 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 09:31:32.161234 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:31:32.163067 21760 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:32.163656 21760 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:32.171078 21760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5dce72fad7fd4471b8bc009390266005: Generated new TSK 0
I20260926 09:31:32.171274 21760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:31:32.175302 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 21678
W20260926 09:31:32.179571 21663 connection.cc:535] client connection to 127.0.0.1:41961 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41961): Connection reset by peer (error 104)
2026-09-26T09:31:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:31:32 dist-test-slave-dfzm krb5kdc[21777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:31:32 dist-test-slave-dfzm krb5kdc[21777](info): set up 2 sockets
Sep 26 09:31:32 dist-test-slave-dfzm krb5kdc[21777](info): commencing operation
krb5kdc: starting...
W20260926 09:31:33.209654 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:31:33 dist-test-slave-dfzm krb5kdc[21777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:31:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:31:33.333444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35811
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:33.418087 21790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:33.418332 21790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:33.418370 21790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:33.419605 21790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:33.419682 21790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:33.419705 21790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:33.419725 21790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:33.419780 21790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:33.421373 21790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35811
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:33.421695 21790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:33.421937 21790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:33.426972 21798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:33.426980 21796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:33.426980 21795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:33.427340 21790 server_base.cc:1034] running on GCE node
I20260926 09:31:33.427556 21790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:33.427836 21790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:33.428989 21790 hybrid_clock.cc:648] HybridClock initialized: now 1790415093428975 us; error 30 us; skew 500 ppm
Sep 26 09:31:33 dist-test-slave-dfzm krb5kdc[21777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:33.431519 21790 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:31:33.432039 21790 webserver.cc:460] Webserver started at http://127.0.0.1:34931/ using document root <none> and password file <none>
I20260926 09:31:33.432286 21790 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:33.432358 21790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:33.432512 21790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:33.433578 21790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "b533fe16192b44eeb73b1aa3b03f432b"
format_stamp: "Formatted at 2026-09-26 09:31:33 on dist-test-slave-dfzm"
server_key: "1ad1bdf4c4cacfe884f4e3e35389b0d9"
server_key_iv: "ff562e4f52052fee4d7f4c49ff1bb153"
server_key_version: "encryptionkey@0"
I20260926 09:31:33.434013 21790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "b533fe16192b44eeb73b1aa3b03f432b"
format_stamp: "Formatted at 2026-09-26 09:31:33 on dist-test-slave-dfzm"
server_key: "1ad1bdf4c4cacfe884f4e3e35389b0d9"
server_key_iv: "ff562e4f52052fee4d7f4c49ff1bb153"
server_key_version: "encryptionkey@0"
I20260926 09:31:33.435379 21790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:33.436165 21805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:33.436374 21790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:33.436476 21790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "b533fe16192b44eeb73b1aa3b03f432b"
format_stamp: "Formatted at 2026-09-26 09:31:33 on dist-test-slave-dfzm"
server_key: "1ad1bdf4c4cacfe884f4e3e35389b0d9"
server_key_iv: "ff562e4f52052fee4d7f4c49ff1bb153"
server_key_version: "encryptionkey@0"
I20260926 09:31:33.436563 21790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:33.440035 21790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:33.442206 21790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:33.442365 21790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:33.446543 21790 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35811
I20260926 09:31:33.446580 21857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35811 every 8 connection(s)
I20260926 09:31:33.446954 21790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:33.447989 21858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:33.448519 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 21790
I20260926 09:31:33.448653 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:33.448838 392 external_mini_cluster.cc:1428] Setting key 30fb97deeee0e5c2aedec9c979a39af3
I20260926 09:31:33.451332 21858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b533fe16192b44eeb73b1aa3b03f432b: Bootstrap starting.
I20260926 09:31:33.451368 21861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:33.449415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:38217 (local address 127.0.0.1:35811)
0926 09:31:33.449618 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:33.449628 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:33.449684 (+ 56us) server_negotiation.cc:407] Connection header received
0926 09:31:33.450092 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:33.450098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:33.450123 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:33.450245 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:33.450902 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:33.450905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:33.450906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:33.450909 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:33.450944 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:33.450955 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:33.451007 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:33.451108 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 09:31:33.451166 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
I20260926 09:31:33.452196 21858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b533fe16192b44eeb73b1aa3b03f432b: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:33.452508 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:31:33.452582 21858 log.cc:824] T 00000000000000000000000000000000 P b533fe16192b44eeb73b1aa3b03f432b: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:33.453413 21858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b533fe16192b44eeb73b1aa3b03f432b: No bootstrap required, opened a new log
I20260926 09:31:33.455256 21861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:33.454231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:48595 (local address 127.0.0.1:35811)
0926 09:31:33.454270 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:33.454272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:33.454341 (+ 69us) server_negotiation.cc:407] Connection header received
0926 09:31:33.454419 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:33.454419 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:33.454423 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:33.454472 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:33.455040 (+ 568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:33.455040 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:33.455041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:33.455044 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:33.455057 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:33.455060 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:33.455099 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:33.455160 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 09:31:33.455172 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:31:33.456312 21858 raft_consensus.cc:348] T 00000000000000000000000000000000 P b533fe16192b44eeb73b1aa3b03f432b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b533fe16192b44eeb73b1aa3b03f432b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35811 } }
I20260926 09:31:33.456493 21858 raft_consensus.cc:374] T 00000000000000000000000000000000 P b533fe16192b44eeb73b1aa3b03f432b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:33.456543 21858 raft_consensus.cc:729] T 00000000000000000000000000000000 P b533fe16192b44eeb73b1aa3b03f432b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b533fe16192b44eeb73b1aa3b03f432b, State: Initialized, Role: FOLLOWER
I20260926 09:31:33.456700 21858 consensus_queue.cc:260] T 00000000000000000000000000000000 P b533fe16192b44eeb73b1aa3b03f432b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b533fe16192b44eeb73b1aa3b03f432b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35811 } }
I20260926 09:31:33.456809 21858 raft_consensus.cc:388] T 00000000000000000000000000000000 P b533fe16192b44eeb73b1aa3b03f432b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:31:33.456859 21858 raft_consensus.cc:482] T 00000000000000000000000000000000 P b533fe16192b44eeb73b1aa3b03f432b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:31:33.456913 21858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b533fe16192b44eeb73b1aa3b03f432b [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:33.457544 21858 raft_consensus.cc:504] T 00000000000000000000000000000000 P b533fe16192b44eeb73b1aa3b03f432b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b533fe16192b44eeb73b1aa3b03f432b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35811 } }
I20260926 09:31:33.457700 21858 leader_election.cc:302] T 00000000000000000000000000000000 P b533fe16192b44eeb73b1aa3b03f432b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b533fe16192b44eeb73b1aa3b03f432b; no voters:
I20260926 09:31:33.457912 21858 leader_election.cc:288] T 00000000000000000000000000000000 P b533fe16192b44eeb73b1aa3b03f432b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:31:33.457970 21868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b533fe16192b44eeb73b1aa3b03f432b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:33.458135 21868 raft_consensus.cc:686] T 00000000000000000000000000000000 P b533fe16192b44eeb73b1aa3b03f432b [term 1 LEADER]: Becoming Leader. State: Replica: b533fe16192b44eeb73b1aa3b03f432b, State: Running, Role: LEADER
I20260926 09:31:33.458206 21858 sys_catalog.cc:564] T 00000000000000000000000000000000 P b533fe16192b44eeb73b1aa3b03f432b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:33.458253 21868 consensus_queue.cc:237] T 00000000000000000000000000000000 P b533fe16192b44eeb73b1aa3b03f432b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b533fe16192b44eeb73b1aa3b03f432b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35811 } }
I20260926 09:31:33.458638 21869 sys_catalog.cc:455] T 00000000000000000000000000000000 P b533fe16192b44eeb73b1aa3b03f432b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b533fe16192b44eeb73b1aa3b03f432b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b533fe16192b44eeb73b1aa3b03f432b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35811 } } }
I20260926 09:31:33.458655 21870 sys_catalog.cc:455] T 00000000000000000000000000000000 P b533fe16192b44eeb73b1aa3b03f432b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b533fe16192b44eeb73b1aa3b03f432b. Latest consensus state: current_term: 1 leader_uuid: "b533fe16192b44eeb73b1aa3b03f432b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b533fe16192b44eeb73b1aa3b03f432b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35811 } } }
I20260926 09:31:33.458768 21869 sys_catalog.cc:458] T 00000000000000000000000000000000 P b533fe16192b44eeb73b1aa3b03f432b [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:33.459028 21874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:33.458774 21870 sys_catalog.cc:458] T 00000000000000000000000000000000 P b533fe16192b44eeb73b1aa3b03f432b [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:33.459641 21874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:33.461364 21874 catalog_manager.cc:1347] Generated new cluster ID: 4261f6234d7f4df384d8d4a9f980c038
I20260926 09:31:33.461447 21874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:31:33.473028 21874 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:33.473249 21874 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:33.479808 21874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b533fe16192b44eeb73b1aa3b03f432b: Generated new TSK 0
I20260926 09:31:33.480041 21874 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:31:33.486799 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 21790
2026-09-26T09:31:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:31:33 dist-test-slave-dfzm krb5kdc[21889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:31:33 dist-test-slave-dfzm krb5kdc[21889](info): set up 2 sockets
Sep 26 09:31:33 dist-test-slave-dfzm krb5kdc[21889](info): commencing operation
krb5kdc: starting...
W20260926 09:31:34.529820 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:31:34 dist-test-slave-dfzm krb5kdc[21889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:31:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:31:34.648162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36559
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:34.727689 21902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:34.727905 21902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:34.727929 21902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:34.729013 21902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:34.729051 21902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:34.729065 21902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:34.729079 21902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:34.729090 21902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:34.730520 21902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36559
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:34.730706 21902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:34.730878 21902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:34.735628 21907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:34.735749 21910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:34.735603 21908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:34.736001 21902 server_base.cc:1034] running on GCE node
I20260926 09:31:34.736272 21902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:34.736541 21902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:34.737691 21902 hybrid_clock.cc:648] HybridClock initialized: now 1790415094737678 us; error 31 us; skew 500 ppm
Sep 26 09:31:34 dist-test-slave-dfzm krb5kdc[21889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:34.740252 21902 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:31:34.740762 21902 webserver.cc:460] Webserver started at http://127.0.98.62:34145/ using document root <none> and password file <none>
I20260926 09:31:34.741021 21902 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:34.741091 21902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:34.741237 21902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:34.742388 21902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "eeaddfaebbef4d62a717372b0b481157"
format_stamp: "Formatted at 2026-09-26 09:31:34 on dist-test-slave-dfzm"
server_key: "e06e79e52fc66624559e9dfada8a0a93"
server_key_iv: "57257319d34a8df958b8a9075d1bc14f"
server_key_version: "encryptionkey@0"
I20260926 09:31:34.742755 21902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "eeaddfaebbef4d62a717372b0b481157"
format_stamp: "Formatted at 2026-09-26 09:31:34 on dist-test-slave-dfzm"
server_key: "e06e79e52fc66624559e9dfada8a0a93"
server_key_iv: "57257319d34a8df958b8a9075d1bc14f"
server_key_version: "encryptionkey@0"
I20260926 09:31:34.744120 21902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:31:34.744943 21917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:34.745153 21902 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:34.745260 21902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "eeaddfaebbef4d62a717372b0b481157"
format_stamp: "Formatted at 2026-09-26 09:31:34 on dist-test-slave-dfzm"
server_key: "e06e79e52fc66624559e9dfada8a0a93"
server_key_iv: "57257319d34a8df958b8a9075d1bc14f"
server_key_version: "encryptionkey@0"
I20260926 09:31:34.745343 21902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:34.758339 21902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:34.760325 21902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:34.760502 21902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:34.764797 21902 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36559
I20260926 09:31:34.764881 21969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36559 every 8 connection(s)
I20260926 09:31:34.765254 21902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:34.766250 21970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:34.768738 21970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeaddfaebbef4d62a717372b0b481157: Bootstrap starting.
I20260926 09:31:34.769428 21970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eeaddfaebbef4d62a717372b0b481157: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:34.769883 21970 log.cc:824] T 00000000000000000000000000000000 P eeaddfaebbef4d62a717372b0b481157: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:34.770773 21970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeaddfaebbef4d62a717372b0b481157: No bootstrap required, opened a new log
I20260926 09:31:34.772445 21970 raft_consensus.cc:348] T 00000000000000000000000000000000 P eeaddfaebbef4d62a717372b0b481157 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeaddfaebbef4d62a717372b0b481157" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36559 } }
I20260926 09:31:34.772599 21970 raft_consensus.cc:374] T 00000000000000000000000000000000 P eeaddfaebbef4d62a717372b0b481157 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:34.772624 21970 raft_consensus.cc:729] T 00000000000000000000000000000000 P eeaddfaebbef4d62a717372b0b481157 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeaddfaebbef4d62a717372b0b481157, State: Initialized, Role: FOLLOWER
I20260926 09:31:34.772787 21970 consensus_queue.cc:260] T 00000000000000000000000000000000 P eeaddfaebbef4d62a717372b0b481157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeaddfaebbef4d62a717372b0b481157" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36559 } }
I20260926 09:31:34.772854 21970 raft_consensus.cc:388] T 00000000000000000000000000000000 P eeaddfaebbef4d62a717372b0b481157 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:31:34.772876 21970 raft_consensus.cc:482] T 00000000000000000000000000000000 P eeaddfaebbef4d62a717372b0b481157 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:31:34.772946 21970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eeaddfaebbef4d62a717372b0b481157 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:34.772953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 21902
I20260926 09:31:34.773079 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:34.773262 392 external_mini_cluster.cc:1428] Setting key ca4453cf05ec4c0e7fb4b7d0f0a020b9
I20260926 09:31:34.773743 21970 raft_consensus.cc:504] T 00000000000000000000000000000000 P eeaddfaebbef4d62a717372b0b481157 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeaddfaebbef4d62a717372b0b481157" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36559 } }
I20260926 09:31:34.773869 21970 leader_election.cc:302] T 00000000000000000000000000000000 P eeaddfaebbef4d62a717372b0b481157 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eeaddfaebbef4d62a717372b0b481157; no voters:
I20260926 09:31:34.774044 21970 leader_election.cc:288] T 00000000000000000000000000000000 P eeaddfaebbef4d62a717372b0b481157 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:31:34.774077 21975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eeaddfaebbef4d62a717372b0b481157 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:34.774220 21975 raft_consensus.cc:686] T 00000000000000000000000000000000 P eeaddfaebbef4d62a717372b0b481157 [term 1 LEADER]: Becoming Leader. State: Replica: eeaddfaebbef4d62a717372b0b481157, State: Running, Role: LEADER
I20260926 09:31:34.774343 21970 sys_catalog.cc:564] T 00000000000000000000000000000000 P eeaddfaebbef4d62a717372b0b481157 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:34.774343 21975 consensus_queue.cc:237] T 00000000000000000000000000000000 P eeaddfaebbef4d62a717372b0b481157 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeaddfaebbef4d62a717372b0b481157" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36559 } }
I20260926 09:31:34.775273 21977 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeaddfaebbef4d62a717372b0b481157 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eeaddfaebbef4d62a717372b0b481157. Latest consensus state: current_term: 1 leader_uuid: "eeaddfaebbef4d62a717372b0b481157" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeaddfaebbef4d62a717372b0b481157" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36559 } } }
I20260926 09:31:34.775434 21977 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeaddfaebbef4d62a717372b0b481157 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:34.775486 21976 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeaddfaebbef4d62a717372b0b481157 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eeaddfaebbef4d62a717372b0b481157" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeaddfaebbef4d62a717372b0b481157" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36559 } } }
I20260926 09:31:34.775547 21976 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeaddfaebbef4d62a717372b0b481157 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:34.775692 21986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:34.776295 21986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:34.777611 21992 mvcc.cc:205] Tried to move back new op lower bound from 7333540228207321088 to 7333540228197130240. Current Snapshot: MvccSnapshot[applied={T|T < 7333540228207321088}]
I20260926 09:31:34.778059 21986 catalog_manager.cc:1347] Generated new cluster ID: cff132e4bc104e6a810701d1558af9c2
I20260926 09:31:34.778119 21986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:31:34 dist-test-slave-dfzm krb5kdc[21889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:31:34.784444 21974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:34.773792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46237 (local address 127.0.98.62:36559)
0926 09:31:34.774072 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:34.774083 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:34.774121 (+ 38us) server_negotiation.cc:407] Connection header received
0926 09:31:34.774248 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:34.774256 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:34.774315 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:34.774498 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:34.775251 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:34.776684 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:34.777484 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:34.778111 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:34.778236 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:34.780869 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:34.780874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:34.780874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:34.780889 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:34.782759 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:34.783180 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:34.783184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:34.783184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:34.783286 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:34.783538 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:34.783539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:34.783540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:34.783806 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:34.783982 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:34.784106 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 09:31:34.784211 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":126,"threads_started":1}
I20260926 09:31:34.785481 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:31:34.792316 21986 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:34.793007 21986 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:34.797201 21974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:34.787037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34395 (local address 127.0.98.62:36559)
0926 09:31:34.787074 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:34.787076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:34.789798 (+ 2722us) server_negotiation.cc:407] Connection header received
0926 09:31:34.789865 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:34.789866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:34.789878 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:34.789937 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:34.790620 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:34.791399 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:34.792310 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:34.792647 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:34.792725 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:34.794238 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:34.794240 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:34.794240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:34.794248 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:34.795904 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:34.796389 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:34.796390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:34.796391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:34.796451 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:34.796685 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:34.796686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:34.796686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:34.796807 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:34.796905 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:34.797046 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 09:31:34.797110 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":18}
I20260926 09:31:34.798550 21986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eeaddfaebbef4d62a717372b0b481157: Generated new TSK 0
I20260926 09:31:34.798738 21986 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:31:34.802203 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 21902
W20260926 09:31:34.807189 21887 connection.cc:535] client connection to 127.0.98.62:36559 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36559): Connection reset by peer (error 104)
2026-09-26T09:31:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:31:34 dist-test-slave-dfzm krb5kdc[22001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:31:34 dist-test-slave-dfzm krb5kdc[22001](info): set up 2 sockets
Sep 26 09:31:34 dist-test-slave-dfzm krb5kdc[22001](info): commencing operation
krb5kdc: starting...
W20260926 09:31:35.829813 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:31:35 dist-test-slave-dfzm krb5kdc[22001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:31:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:31:35.948777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33903
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:36.031993 22014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:36.032223 22014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:36.032248 22014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:36.033564 22014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:36.033632 22014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:36.033668 22014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:36.033705 22014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:36.033733 22014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:36.035288 22014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33903
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:36.035568 22014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:36.035815 22014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:36.040889 22022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:36.040926 22020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:36.041114 22014 server_base.cc:1034] running on GCE node
W20260926 09:31:36.040899 22019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:36.041468 22014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:36.041807 22014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:36.042980 22014 hybrid_clock.cc:648] HybridClock initialized: now 1790415096042972 us; error 39 us; skew 500 ppm
Sep 26 09:31:36 dist-test-slave-dfzm krb5kdc[22001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:36.049233 22014 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:31:36.049780 22014 webserver.cc:460] Webserver started at http://127.0.98.62:36099/ using document root <none> and password file <none>
I20260926 09:31:36.050010 22014 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:36.050077 22014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:36.050235 22014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:36.051319 22014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "6a0cf3e24a0e46e4b137e75d6e19138b"
format_stamp: "Formatted at 2026-09-26 09:31:36 on dist-test-slave-dfzm"
server_key: "58c9641f69131b48fbab87e8f752475c"
server_key_iv: "5821572acf8e1ecb5cb7a3c1ff1e2dd4"
server_key_version: "encryptionkey@0"
I20260926 09:31:36.051697 22014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "6a0cf3e24a0e46e4b137e75d6e19138b"
format_stamp: "Formatted at 2026-09-26 09:31:36 on dist-test-slave-dfzm"
server_key: "58c9641f69131b48fbab87e8f752475c"
server_key_iv: "5821572acf8e1ecb5cb7a3c1ff1e2dd4"
server_key_version: "encryptionkey@0"
I20260926 09:31:36.053038 22014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:31:36.053864 22029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:36.054028 22014 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:31:36.054126 22014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "6a0cf3e24a0e46e4b137e75d6e19138b"
format_stamp: "Formatted at 2026-09-26 09:31:36 on dist-test-slave-dfzm"
server_key: "58c9641f69131b48fbab87e8f752475c"
server_key_iv: "5821572acf8e1ecb5cb7a3c1ff1e2dd4"
server_key_version: "encryptionkey@0"
I20260926 09:31:36.054209 22014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:36.068718 22014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:36.071309 22014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:36.071492 22014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:36.075573 22014 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33903
I20260926 09:31:36.075613 22081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33903 every 8 connection(s)
I20260926 09:31:36.075997 22014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:36.076874 22082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:36.079272 22082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a0cf3e24a0e46e4b137e75d6e19138b: Bootstrap starting.
I20260926 09:31:36.079873 22082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a0cf3e24a0e46e4b137e75d6e19138b: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:36.080168 22082 log.cc:824] T 00000000000000000000000000000000 P 6a0cf3e24a0e46e4b137e75d6e19138b: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:36.080866 22082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a0cf3e24a0e46e4b137e75d6e19138b: No bootstrap required, opened a new log
I20260926 09:31:36.082562 22082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a0cf3e24a0e46e4b137e75d6e19138b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a0cf3e24a0e46e4b137e75d6e19138b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33903 } }
I20260926 09:31:36.082705 22082 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a0cf3e24a0e46e4b137e75d6e19138b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:36.082727 22082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a0cf3e24a0e46e4b137e75d6e19138b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0cf3e24a0e46e4b137e75d6e19138b, State: Initialized, Role: FOLLOWER
I20260926 09:31:36.082911 22082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a0cf3e24a0e46e4b137e75d6e19138b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a0cf3e24a0e46e4b137e75d6e19138b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33903 } }
I20260926 09:31:36.082978 22082 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a0cf3e24a0e46e4b137e75d6e19138b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:31:36.083019 22082 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a0cf3e24a0e46e4b137e75d6e19138b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:31:36.083069 22082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a0cf3e24a0e46e4b137e75d6e19138b [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:36.083623 22082 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a0cf3e24a0e46e4b137e75d6e19138b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a0cf3e24a0e46e4b137e75d6e19138b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33903 } }
I20260926 09:31:36.083740 22082 leader_election.cc:302] T 00000000000000000000000000000000 P 6a0cf3e24a0e46e4b137e75d6e19138b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a0cf3e24a0e46e4b137e75d6e19138b; no voters:
I20260926 09:31:36.083943 22082 leader_election.cc:288] T 00000000000000000000000000000000 P 6a0cf3e24a0e46e4b137e75d6e19138b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:31:36.084051 22085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a0cf3e24a0e46e4b137e75d6e19138b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:36.084198 22082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a0cf3e24a0e46e4b137e75d6e19138b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:36.084255 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 22014
I20260926 09:31:36.084223 22085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a0cf3e24a0e46e4b137e75d6e19138b [term 1 LEADER]: Becoming Leader. State: Replica: 6a0cf3e24a0e46e4b137e75d6e19138b, State: Running, Role: LEADER
I20260926 09:31:36.084429 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:36.084522 22085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a0cf3e24a0e46e4b137e75d6e19138b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a0cf3e24a0e46e4b137e75d6e19138b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33903 } }
I20260926 09:31:36.084643 392 external_mini_cluster.cc:1428] Setting key 72e34e3543393162d181adc2dd786d76
I20260926 09:31:36.084913 22087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a0cf3e24a0e46e4b137e75d6e19138b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a0cf3e24a0e46e4b137e75d6e19138b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a0cf3e24a0e46e4b137e75d6e19138b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33903 } } }
I20260926 09:31:36.085018 22087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a0cf3e24a0e46e4b137e75d6e19138b [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:36.085062 22086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a0cf3e24a0e46e4b137e75d6e19138b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a0cf3e24a0e46e4b137e75d6e19138b. Latest consensus state: current_term: 1 leader_uuid: "6a0cf3e24a0e46e4b137e75d6e19138b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a0cf3e24a0e46e4b137e75d6e19138b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33903 } } }
I20260926 09:31:36.085146 22086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a0cf3e24a0e46e4b137e75d6e19138b [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:36.085318 22095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:36.085896 22095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:36.087473 22095 catalog_manager.cc:1347] Generated new cluster ID: 3065070ce4aa44caa3ed33b78691c85b
I20260926 09:31:36.087534 22095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:31:36.091493 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:36.085288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60729 (local address 127.0.98.62:33903)
0926 09:31:36.085717 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:36.085726 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:36.085755 (+ 29us) server_negotiation.cc:407] Connection header received
0926 09:31:36.085873 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:36.085882 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:36.085941 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:36.086077 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:36.086437 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:36.087818 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:36.088937 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:36.089615 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:36.089768 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:36.090753 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:36.090758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:36.090758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:36.090968 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:36.091008 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:36.091022 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:36.091101 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:36.091237 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 09:31:36.091266 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":79,"threads_started":1}
I20260926 09:31:36.092479 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:31:36.097802 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:36.094208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60267 (local address 127.0.98.62:33903)
0926 09:31:36.094239 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:36.094241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:36.094249 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:31:36.094314 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:36.094315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:36.094323 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:36.094368 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:36.094505 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:36.095233 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:36.096296 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:36.096609 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:36.096672 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:36.097457 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:36.097458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:36.097458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:36.097464 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:36.097478 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:36.097483 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:36.097541 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:36.097691 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 09:31:36.097710 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 09:31:36.101470 22095 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:36.102015 22095 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:36.111966 22095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a0cf3e24a0e46e4b137e75d6e19138b: Generated new TSK 0
I20260926 09:31:36.112195 22095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:31:36.117002 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 22014
W20260926 09:31:36.121269 21999 connection.cc:535] client connection to 127.0.98.62:33903 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:33903): Connection reset by peer (error 104)
2026-09-26T09:31:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:31:36 dist-test-slave-dfzm krb5kdc[22113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:31:36 dist-test-slave-dfzm krb5kdc[22113](info): set up 2 sockets
Sep 26 09:31:36 dist-test-slave-dfzm krb5kdc[22113](info): commencing operation
krb5kdc: starting...
W20260926 09:31:37.157795 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:31:37 dist-test-slave-dfzm krb5kdc[22113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:31:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:31:37.277230 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33515
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:37.358703 22126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:37.358920 22126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:37.358943 22126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:37.360044 22126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:37.360081 22126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:37.360095 22126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:37.360108 22126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:37.360121 22126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:37.361502 22126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33515
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:37.361774 22126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:37.361963 22126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:37.366837 22132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:37.366824 22134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:37.366827 22131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:37.367111 22126 server_base.cc:1034] running on GCE node
I20260926 09:31:37.367364 22126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:37.367623 22126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:37.368777 22126 hybrid_clock.cc:648] HybridClock initialized: now 1790415097368758 us; error 33 us; skew 500 ppm
Sep 26 09:31:37 dist-test-slave-dfzm krb5kdc[22113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:37.371441 22126 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:31:37.371893 22126 webserver.cc:460] Webserver started at http://127.0.98.62:42657/ using document root <none> and password file <none>
I20260926 09:31:37.372118 22126 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:37.372184 22126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:37.372336 22126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:37.373853 22126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "c155000fef5d436bada687dc0b6da80b"
format_stamp: "Formatted at 2026-09-26 09:31:37 on dist-test-slave-dfzm"
server_key: "e697c435c6dfa2536d4bbab1cd87ca42"
server_key_iv: "682be8046973c11f1570fc17c6bc8da8"
server_key_version: "encryptionkey@0"
I20260926 09:31:37.374253 22126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "c155000fef5d436bada687dc0b6da80b"
format_stamp: "Formatted at 2026-09-26 09:31:37 on dist-test-slave-dfzm"
server_key: "e697c435c6dfa2536d4bbab1cd87ca42"
server_key_iv: "682be8046973c11f1570fc17c6bc8da8"
server_key_version: "encryptionkey@0"
I20260926 09:31:37.375727 22126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:31:37.376577 22141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:37.376794 22126 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:37.376900 22126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "c155000fef5d436bada687dc0b6da80b"
format_stamp: "Formatted at 2026-09-26 09:31:37 on dist-test-slave-dfzm"
server_key: "e697c435c6dfa2536d4bbab1cd87ca42"
server_key_iv: "682be8046973c11f1570fc17c6bc8da8"
server_key_version: "encryptionkey@0"
I20260926 09:31:37.376986 22126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:37.380374 22126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:37.382639 22126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:37.382800 22126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:37.386744 22126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33515
I20260926 09:31:37.386797 22193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33515 every 8 connection(s)
I20260926 09:31:37.387144 22126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:37.388134 22194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:37.390437 22194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c155000fef5d436bada687dc0b6da80b: Bootstrap starting.
I20260926 09:31:37.391005 22194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c155000fef5d436bada687dc0b6da80b: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:37.391290 22194 log.cc:824] T 00000000000000000000000000000000 P c155000fef5d436bada687dc0b6da80b: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:37.391810 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 22126
I20260926 09:31:37.391937 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:37.392100 392 external_mini_cluster.cc:1428] Setting key ccbdee1fecf588794761909be7ade068
I20260926 09:31:37.392097 22194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c155000fef5d436bada687dc0b6da80b: No bootstrap required, opened a new log
I20260926 09:31:37.394438 22194 raft_consensus.cc:348] T 00000000000000000000000000000000 P c155000fef5d436bada687dc0b6da80b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c155000fef5d436bada687dc0b6da80b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33515 } }
I20260926 09:31:37.394618 22194 raft_consensus.cc:374] T 00000000000000000000000000000000 P c155000fef5d436bada687dc0b6da80b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:37.394665 22194 raft_consensus.cc:729] T 00000000000000000000000000000000 P c155000fef5d436bada687dc0b6da80b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c155000fef5d436bada687dc0b6da80b, State: Initialized, Role: FOLLOWER
I20260926 09:31:37.394936 22198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:37.392608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44179 (local address 127.0.98.62:33515)
0926 09:31:37.392782 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:37.392791 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:37.392826 (+ 35us) server_negotiation.cc:407] Connection header received
0926 09:31:37.393396 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:37.393403 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:37.393435 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:37.393548 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:37.394229 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:37.394232 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:37.394233 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:37.394236 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:37.394272 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:37.394282 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:37.394337 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:37.394425 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 09:31:37.394482 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260926 09:31:37.394879 22194 consensus_queue.cc:260] T 00000000000000000000000000000000 P c155000fef5d436bada687dc0b6da80b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c155000fef5d436bada687dc0b6da80b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33515 } }
I20260926 09:31:37.395054 22194 raft_consensus.cc:388] T 00000000000000000000000000000000 P c155000fef5d436bada687dc0b6da80b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:31:37.395123 22194 raft_consensus.cc:482] T 00000000000000000000000000000000 P c155000fef5d436bada687dc0b6da80b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:31:37.395174 22194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c155000fef5d436bada687dc0b6da80b [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:37.395816 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:31:37.395805 22194 raft_consensus.cc:504] T 00000000000000000000000000000000 P c155000fef5d436bada687dc0b6da80b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c155000fef5d436bada687dc0b6da80b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33515 } }
I20260926 09:31:37.395905 22194 leader_election.cc:302] T 00000000000000000000000000000000 P c155000fef5d436bada687dc0b6da80b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c155000fef5d436bada687dc0b6da80b; no voters:
I20260926 09:31:37.396052 22194 leader_election.cc:288] T 00000000000000000000000000000000 P c155000fef5d436bada687dc0b6da80b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:31:37.396111 22199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c155000fef5d436bada687dc0b6da80b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:37.396273 22194 sys_catalog.cc:564] T 00000000000000000000000000000000 P c155000fef5d436bada687dc0b6da80b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:37.396284 22199 raft_consensus.cc:686] T 00000000000000000000000000000000 P c155000fef5d436bada687dc0b6da80b [term 1 LEADER]: Becoming Leader. State: Replica: c155000fef5d436bada687dc0b6da80b, State: Running, Role: LEADER
I20260926 09:31:37.396423 22199 consensus_queue.cc:237] T 00000000000000000000000000000000 P c155000fef5d436bada687dc0b6da80b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c155000fef5d436bada687dc0b6da80b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33515 } }
I20260926 09:31:37.396742 22199 sys_catalog.cc:455] T 00000000000000000000000000000000 P c155000fef5d436bada687dc0b6da80b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c155000fef5d436bada687dc0b6da80b. Latest consensus state: current_term: 1 leader_uuid: "c155000fef5d436bada687dc0b6da80b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c155000fef5d436bada687dc0b6da80b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33515 } } }
I20260926 09:31:37.396979 22199 sys_catalog.cc:458] T 00000000000000000000000000000000 P c155000fef5d436bada687dc0b6da80b [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:37.396896 22200 sys_catalog.cc:455] T 00000000000000000000000000000000 P c155000fef5d436bada687dc0b6da80b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c155000fef5d436bada687dc0b6da80b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c155000fef5d436bada687dc0b6da80b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33515 } } }
I20260926 09:31:37.397384 22200 sys_catalog.cc:458] T 00000000000000000000000000000000 P c155000fef5d436bada687dc0b6da80b [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:37.397504 22212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:37.398133 22212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:37.399904 22212 catalog_manager.cc:1347] Generated new cluster ID: f843a739116b488d89fd2a266ed0d25c
I20260926 09:31:37.399961 22212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:31:37.401748 22198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:37.398090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59863 (local address 127.0.98.62:33515)
0926 09:31:37.398124 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:37.398126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:37.398854 (+ 728us) server_negotiation.cc:407] Connection header received
0926 09:31:37.398903 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:37.398903 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:37.398908 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:37.398953 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:37.399677 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:37.399679 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:37.399679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:37.399684 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:37.399700 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:37.399704 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:37.399792 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:37.399895 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 09:31:37.399911 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 09:31:37.423240 22212 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:37.423456 22212 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:37.430451 22212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c155000fef5d436bada687dc0b6da80b: Generated new TSK 0
I20260926 09:31:37.430684 22212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:31:37.436728 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 22126
2026-09-26T09:31:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:31:37 dist-test-slave-dfzm krb5kdc[22225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:31:37 dist-test-slave-dfzm krb5kdc[22225](info): set up 2 sockets
Sep 26 09:31:37 dist-test-slave-dfzm krb5kdc[22225](info): commencing operation
krb5kdc: starting...
W20260926 09:31:38.501749 392 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:31:38 dist-test-slave-dfzm krb5kdc[22225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:31:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:31:38.621423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43089
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:38.702759 22238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:38.702967 22238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:38.702991 22238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:38.704201 22238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:38.704250 22238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:38.704265 22238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:38.704278 22238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:38.704290 22238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:38.705803 22238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:38.706064 22238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:38.706295 22238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:38.711323 22246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:38.711328 22243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:38.711480 22238 server_base.cc:1034] running on GCE node
W20260926 09:31:38.711397 22244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:38.711892 22238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:38.712222 22238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:38.713404 22238 hybrid_clock.cc:648] HybridClock initialized: now 1790415098713391 us; error 45 us; skew 500 ppm
Sep 26 09:31:38 dist-test-slave-dfzm krb5kdc[22225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:38.715956 22238 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:31:38.716470 22238 webserver.cc:460] Webserver started at http://127.0.98.62:44245/ using document root <none> and password file <none>
I20260926 09:31:38.716706 22238 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:38.716773 22238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:38.716912 22238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:38.718075 22238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "937cb25836434425ae7bddcb6f5017e4"
format_stamp: "Formatted at 2026-09-26 09:31:38 on dist-test-slave-dfzm"
server_key: "3088b19a517fc861bf3af30dd727bd1c"
server_key_iv: "b9bc62e57e97ca481db41c82c482a68f"
server_key_version: "encryptionkey@0"
I20260926 09:31:38.718441 22238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "937cb25836434425ae7bddcb6f5017e4"
format_stamp: "Formatted at 2026-09-26 09:31:38 on dist-test-slave-dfzm"
server_key: "3088b19a517fc861bf3af30dd727bd1c"
server_key_iv: "b9bc62e57e97ca481db41c82c482a68f"
server_key_version: "encryptionkey@0"
I20260926 09:31:38.719823 22238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:31:38.720724 22253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:38.720995 22238 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:38.721112 22238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "937cb25836434425ae7bddcb6f5017e4"
format_stamp: "Formatted at 2026-09-26 09:31:38 on dist-test-slave-dfzm"
server_key: "3088b19a517fc861bf3af30dd727bd1c"
server_key_iv: "b9bc62e57e97ca481db41c82c482a68f"
server_key_version: "encryptionkey@0"
I20260926 09:31:38.721204 22238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:38.736796 22238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:38.738832 22238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:38.739008 22238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:38.743100 22238 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43089
I20260926 09:31:38.743136 22305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43089 every 8 connection(s)
I20260926 09:31:38.743497 22238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:38.744370 22306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:38.746332 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 22238
I20260926 09:31:38.746462 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:38.746634 392 external_mini_cluster.cc:1428] Setting key 1aa29bb07b55e24b9510d927fd0d9736
I20260926 09:31:38.747120 22306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 937cb25836434425ae7bddcb6f5017e4: Bootstrap starting.
I20260926 09:31:38.747788 22306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 937cb25836434425ae7bddcb6f5017e4: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:38.748085 22306 log.cc:824] T 00000000000000000000000000000000 P 937cb25836434425ae7bddcb6f5017e4: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:38.748869 22306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 937cb25836434425ae7bddcb6f5017e4: No bootstrap required, opened a new log
I20260926 09:31:38.751099 22306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 937cb25836434425ae7bddcb6f5017e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937cb25836434425ae7bddcb6f5017e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43089 } }
I20260926 09:31:38.751268 22306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 937cb25836434425ae7bddcb6f5017e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:38.751293 22306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 937cb25836434425ae7bddcb6f5017e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 937cb25836434425ae7bddcb6f5017e4, State: Initialized, Role: FOLLOWER
I20260926 09:31:38.751432 22306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 937cb25836434425ae7bddcb6f5017e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937cb25836434425ae7bddcb6f5017e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43089 } }
I20260926 09:31:38.751497 22306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 937cb25836434425ae7bddcb6f5017e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:31:38.751518 22306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 937cb25836434425ae7bddcb6f5017e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:31:38.751583 22306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 937cb25836434425ae7bddcb6f5017e4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:38.752365 22306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 937cb25836434425ae7bddcb6f5017e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937cb25836434425ae7bddcb6f5017e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43089 } }
I20260926 09:31:38.752501 22306 leader_election.cc:302] T 00000000000000000000000000000000 P 937cb25836434425ae7bddcb6f5017e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 937cb25836434425ae7bddcb6f5017e4; no voters:
I20260926 09:31:38.752674 22306 leader_election.cc:288] T 00000000000000000000000000000000 P 937cb25836434425ae7bddcb6f5017e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:31:38.752774 22311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 937cb25836434425ae7bddcb6f5017e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:38.752954 22311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 937cb25836434425ae7bddcb6f5017e4 [term 1 LEADER]: Becoming Leader. State: Replica: 937cb25836434425ae7bddcb6f5017e4, State: Running, Role: LEADER
I20260926 09:31:38.752979 22306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 937cb25836434425ae7bddcb6f5017e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:38.753105 22311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 937cb25836434425ae7bddcb6f5017e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937cb25836434425ae7bddcb6f5017e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43089 } }
I20260926 09:31:38.753438 22313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 937cb25836434425ae7bddcb6f5017e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "937cb25836434425ae7bddcb6f5017e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937cb25836434425ae7bddcb6f5017e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43089 } } }
I20260926 09:31:38.753510 22313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 937cb25836434425ae7bddcb6f5017e4 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:38.753767 22317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:38.754107 22312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 937cb25836434425ae7bddcb6f5017e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 937cb25836434425ae7bddcb6f5017e4. Latest consensus state: current_term: 1 leader_uuid: "937cb25836434425ae7bddcb6f5017e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937cb25836434425ae7bddcb6f5017e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43089 } } }
I20260926 09:31:38.754285 22312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 937cb25836434425ae7bddcb6f5017e4 [sys.catalog]: This master's current role is: LEADER
Sep 26 09:31:38 dist-test-slave-dfzm krb5kdc[22225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:31:38.754518 22317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:38.756184 22317 catalog_manager.cc:1347] Generated new cluster ID: 4d462a18161f47a2bd0d02bb2fe62014
I20260926 09:31:38.756232 22317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:31:38.758701 22309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:38.747213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57879 (local address 127.0.98.62:43089)
0926 09:31:38.747419 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:38.747427 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:38.747454 (+ 27us) server_negotiation.cc:407] Connection header received
0926 09:31:38.747969 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:38.747977 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:38.748027 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:38.748147 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:38.748977 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:38.749951 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:38.751121 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:38.751750 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:38.751839 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:38.755056 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:38.755060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:38.755061 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:38.755073 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:38.757013 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:38.757413 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:38.757416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:38.757417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:38.757536 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:38.757800 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:38.757802 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:38.757802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:38.758047 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:38.758227 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:38.758343 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 09:31:38.758490 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":112,"threads_started":1}
I20260926 09:31:38.762585 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:31:38.775983 22309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:38.764010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46749 (local address 127.0.98.62:43089)
0926 09:31:38.764050 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:38.764052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:38.769216 (+ 5164us) server_negotiation.cc:407] Connection header received
0926 09:31:38.769285 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:38.769286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:38.769301 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:38.769382 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:38.770330 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:38.770838 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:38.771566 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:38.771803 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:38.771895 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:38.773272 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:38.773274 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:38.773274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:38.773282 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:38.774822 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:38.775187 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:38.775188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:38.775189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:38.775254 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:38.775462 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:38.775463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:38.775463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:38.775576 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:38.775677 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:38.775788 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 09:31:38.775870 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:31:38.777295 22317 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:38.778081 22317 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:38.791771 22317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 937cb25836434425ae7bddcb6f5017e4: Generated new TSK 0
I20260926 09:31:38.792026 22317 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:31:38.795215 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 22238
2026-09-26T09:31:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:31:38 dist-test-slave-dfzm krb5kdc[22337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:31:38 dist-test-slave-dfzm krb5kdc[22337](info): set up 2 sockets
Sep 26 09:31:38 dist-test-slave-dfzm krb5kdc[22337](info): commencing operation
krb5kdc: starting...
W20260926 09:31:39.829559 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:31:39 dist-test-slave-dfzm krb5kdc[22337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:31:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:31:39.946388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44333
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:40.027530 22350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:40.027760 22350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:40.027786 22350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:40.028931 22350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:40.028970 22350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:40.028985 22350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:40.028997 22350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:40.029008 22350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:40.030483 22350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:40.030754 22350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:40.031018 22350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:40.036005 22358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:40.035988 22356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:40.035988 22355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:40.036429 22350 server_base.cc:1034] running on GCE node
I20260926 09:31:40.036741 22350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:40.037047 22350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:40.038213 22350 hybrid_clock.cc:648] HybridClock initialized: now 1790415100038180 us; error 62 us; skew 500 ppm
Sep 26 09:31:40 dist-test-slave-dfzm krb5kdc[22337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:40.040799 22350 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:31:40.041242 22350 webserver.cc:460] Webserver started at http://127.0.98.62:35647/ using document root <none> and password file <none>
I20260926 09:31:40.041491 22350 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:40.041559 22350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:40.041772 22350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:40.042866 22350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "68a23cc6cc634be6beba5a44a87e37c8"
format_stamp: "Formatted at 2026-09-26 09:31:40 on dist-test-slave-dfzm"
server_key: "e0372f8e1477797322328859f706ee43"
server_key_iv: "157e0c343af881d5d203b701af5cb113"
server_key_version: "encryptionkey@0"
I20260926 09:31:40.043229 22350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "68a23cc6cc634be6beba5a44a87e37c8"
format_stamp: "Formatted at 2026-09-26 09:31:40 on dist-test-slave-dfzm"
server_key: "e0372f8e1477797322328859f706ee43"
server_key_iv: "157e0c343af881d5d203b701af5cb113"
server_key_version: "encryptionkey@0"
I20260926 09:31:40.044606 22350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:31:40.045388 22365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:40.045557 22350 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:31:40.045702 22350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "68a23cc6cc634be6beba5a44a87e37c8"
format_stamp: "Formatted at 2026-09-26 09:31:40 on dist-test-slave-dfzm"
server_key: "e0372f8e1477797322328859f706ee43"
server_key_iv: "157e0c343af881d5d203b701af5cb113"
server_key_version: "encryptionkey@0"
I20260926 09:31:40.045794 22350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:40.052604 22350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:40.054720 22350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:40.054889 22350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:40.058808 22350 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44333
I20260926 09:31:40.058856 22417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44333 every 8 connection(s)
I20260926 09:31:40.059243 22350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:40.060173 22418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:40.061123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 22350
I20260926 09:31:40.061241 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:40.061411 392 external_mini_cluster.cc:1428] Setting key ca1d05a43e5d53590818a273dd2cc469
I20260926 09:31:40.063022 22418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68a23cc6cc634be6beba5a44a87e37c8: Bootstrap starting.
I20260926 09:31:40.063684 22418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68a23cc6cc634be6beba5a44a87e37c8: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:40.063989 22418 log.cc:824] T 00000000000000000000000000000000 P 68a23cc6cc634be6beba5a44a87e37c8: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:40.064718 22418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68a23cc6cc634be6beba5a44a87e37c8: No bootstrap required, opened a new log
I20260926 09:31:40.066864 22418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68a23cc6cc634be6beba5a44a87e37c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68a23cc6cc634be6beba5a44a87e37c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44333 } }
I20260926 09:31:40.067039 22418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68a23cc6cc634be6beba5a44a87e37c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:40.067111 22418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68a23cc6cc634be6beba5a44a87e37c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a23cc6cc634be6beba5a44a87e37c8, State: Initialized, Role: FOLLOWER
I20260926 09:31:40.067312 22421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:40.061993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52641 (local address 127.0.98.62:44333)
0926 09:31:40.062208 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:40.062215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:40.062240 (+ 25us) server_negotiation.cc:407] Connection header received
0926 09:31:40.062798 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:40.062804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:40.062852 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:40.062971 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:40.063255 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:40.064233 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:40.065127 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:40.065543 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:40.065780 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:40.066632 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:40.066635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:40.066636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:40.066797 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:40.066827 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:40.066838 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:40.066911 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:40.066985 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 09:31:40.067049 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":123,"threads_started":1}
I20260926 09:31:40.067319 22418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68a23cc6cc634be6beba5a44a87e37c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68a23cc6cc634be6beba5a44a87e37c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44333 } }
I20260926 09:31:40.067488 22418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68a23cc6cc634be6beba5a44a87e37c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:31:40.067524 22418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68a23cc6cc634be6beba5a44a87e37c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:31:40.067577 22418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68a23cc6cc634be6beba5a44a87e37c8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:40.068181 22418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68a23cc6cc634be6beba5a44a87e37c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68a23cc6cc634be6beba5a44a87e37c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44333 } }
I20260926 09:31:40.068311 22418 leader_election.cc:302] T 00000000000000000000000000000000 P 68a23cc6cc634be6beba5a44a87e37c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68a23cc6cc634be6beba5a44a87e37c8; no voters:
I20260926 09:31:40.068339 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:31:40.068557 22418 leader_election.cc:288] T 00000000000000000000000000000000 P 68a23cc6cc634be6beba5a44a87e37c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:31:40.068591 22423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68a23cc6cc634be6beba5a44a87e37c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:40.068737 22423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68a23cc6cc634be6beba5a44a87e37c8 [term 1 LEADER]: Becoming Leader. State: Replica: 68a23cc6cc634be6beba5a44a87e37c8, State: Running, Role: LEADER
I20260926 09:31:40.068835 22418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68a23cc6cc634be6beba5a44a87e37c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:40.068885 22423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68a23cc6cc634be6beba5a44a87e37c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68a23cc6cc634be6beba5a44a87e37c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44333 } }
I20260926 09:31:40.069325 22423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68a23cc6cc634be6beba5a44a87e37c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68a23cc6cc634be6beba5a44a87e37c8. Latest consensus state: current_term: 1 leader_uuid: "68a23cc6cc634be6beba5a44a87e37c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68a23cc6cc634be6beba5a44a87e37c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44333 } } }
I20260926 09:31:40.069657 22423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68a23cc6cc634be6beba5a44a87e37c8 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:40.069540 22424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68a23cc6cc634be6beba5a44a87e37c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68a23cc6cc634be6beba5a44a87e37c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68a23cc6cc634be6beba5a44a87e37c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44333 } } }
I20260926 09:31:40.070124 22424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68a23cc6cc634be6beba5a44a87e37c8 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:40.070187 22436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:40.070721 22436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:40.072238 22436 catalog_manager.cc:1347] Generated new cluster ID: 87df9803c7db46898fb55012b84fbe1f
I20260926 09:31:40.072288 22436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:31:40.075179 22421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:40.070777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54239 (local address 127.0.98.62:44333)
0926 09:31:40.070816 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:40.070819 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:40.071727 (+ 908us) server_negotiation.cc:407] Connection header received
0926 09:31:40.071798 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:40.071799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:40.071810 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:40.071866 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:40.072028 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:40.072699 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:40.073788 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:40.074074 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:40.074164 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:40.074918 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:40.074920 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:40.074920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:40.074925 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:40.074944 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:40.074949 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:40.075009 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:40.075070 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 09:31:40.075095 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":19}
I20260926 09:31:40.081432 22436 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:40.082307 22436 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:40.085785 22436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68a23cc6cc634be6beba5a44a87e37c8: Generated new TSK 0
I20260926 09:31:40.085968 22436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:31:40.090240 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 22350
2026-09-26T09:31:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:31:40 dist-test-slave-dfzm krb5kdc[22449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:31:40 dist-test-slave-dfzm krb5kdc[22449](info): set up 2 sockets
Sep 26 09:31:40 dist-test-slave-dfzm krb5kdc[22449](info): commencing operation
krb5kdc: starting...
W20260926 09:31:41.129807 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:31:41 dist-test-slave-dfzm krb5kdc[22449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:31:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:31:41.248692 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34083
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:41.329766 22462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:41.329993 22462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:41.330018 22462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:41.331142 22462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:41.331180 22462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:41.331195 22462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:41.331207 22462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:41.331220 22462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:41.332679 22462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:41.332944 22462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:41.333226 22462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:41.338682 22470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:41.338677 22467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:41.338685 22468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:41.338795 22462 server_base.cc:1034] running on GCE node
I20260926 09:31:41.339205 22462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:41.339591 22462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:41.340746 22462 hybrid_clock.cc:648] HybridClock initialized: now 1790415101340720 us; error 49 us; skew 500 ppm
Sep 26 09:31:41 dist-test-slave-dfzm krb5kdc[22449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:41.347952 22462 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:31:41.348484 22462 webserver.cc:460] Webserver started at http://127.0.98.62:36629/ using document root <none> and password file <none>
I20260926 09:31:41.348747 22462 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:41.348831 22462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:41.348992 22462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:41.350154 22462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "0fcae760e36640e38c85f45c6b895378"
format_stamp: "Formatted at 2026-09-26 09:31:41 on dist-test-slave-dfzm"
server_key: "2bbb02c1a8adad93daad2b2ec94af429"
server_key_iv: "b9a705fca75a5c5c4d6bc181803b4c17"
server_key_version: "encryptionkey@0"
I20260926 09:31:41.350528 22462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "0fcae760e36640e38c85f45c6b895378"
format_stamp: "Formatted at 2026-09-26 09:31:41 on dist-test-slave-dfzm"
server_key: "2bbb02c1a8adad93daad2b2ec94af429"
server_key_iv: "b9a705fca75a5c5c4d6bc181803b4c17"
server_key_version: "encryptionkey@0"
I20260926 09:31:41.352191 22462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:31:41.353031 22477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:41.353258 22462 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:41.353370 22462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "0fcae760e36640e38c85f45c6b895378"
format_stamp: "Formatted at 2026-09-26 09:31:41 on dist-test-slave-dfzm"
server_key: "2bbb02c1a8adad93daad2b2ec94af429"
server_key_iv: "b9a705fca75a5c5c4d6bc181803b4c17"
server_key_version: "encryptionkey@0"
I20260926 09:31:41.353451 22462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:41.356824 22462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:41.358872 22462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:41.359040 22462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:41.363055 22462 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34083
I20260926 09:31:41.363085 22529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34083 every 8 connection(s)
I20260926 09:31:41.363415 22462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:41.364378 22530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:41.366653 22530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fcae760e36640e38c85f45c6b895378: Bootstrap starting.
I20260926 09:31:41.367173 22530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fcae760e36640e38c85f45c6b895378: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:41.367411 22530 log.cc:824] T 00000000000000000000000000000000 P 0fcae760e36640e38c85f45c6b895378: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:41.368041 22530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fcae760e36640e38c85f45c6b895378: No bootstrap required, opened a new log
I20260926 09:31:41.369635 22530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fcae760e36640e38c85f45c6b895378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fcae760e36640e38c85f45c6b895378" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34083 } }
I20260926 09:31:41.369805 22530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fcae760e36640e38c85f45c6b895378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:31:41.369830 22530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fcae760e36640e38c85f45c6b895378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fcae760e36640e38c85f45c6b895378, State: Initialized, Role: FOLLOWER
I20260926 09:31:41.369933 22530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fcae760e36640e38c85f45c6b895378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fcae760e36640e38c85f45c6b895378" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34083 } }
I20260926 09:31:41.370011 22530 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fcae760e36640e38c85f45c6b895378 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:31:41.370048 22530 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fcae760e36640e38c85f45c6b895378 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:31:41.370122 22530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fcae760e36640e38c85f45c6b895378 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:41.370652 22530 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fcae760e36640e38c85f45c6b895378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fcae760e36640e38c85f45c6b895378" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34083 } }
I20260926 09:31:41.370772 22530 leader_election.cc:302] T 00000000000000000000000000000000 P 0fcae760e36640e38c85f45c6b895378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fcae760e36640e38c85f45c6b895378; no voters:
I20260926 09:31:41.370954 22530 leader_election.cc:288] T 00000000000000000000000000000000 P 0fcae760e36640e38c85f45c6b895378 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:31:41.371007 22533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fcae760e36640e38c85f45c6b895378 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:41.371196 22533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fcae760e36640e38c85f45c6b895378 [term 1 LEADER]: Becoming Leader. State: Replica: 0fcae760e36640e38c85f45c6b895378, State: Running, Role: LEADER
I20260926 09:31:41.371214 22530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fcae760e36640e38c85f45c6b895378 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:41.371289 22533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fcae760e36640e38c85f45c6b895378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fcae760e36640e38c85f45c6b895378" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34083 } }
I20260926 09:31:41.371939 22534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fcae760e36640e38c85f45c6b895378 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fcae760e36640e38c85f45c6b895378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fcae760e36640e38c85f45c6b895378" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34083 } } }
I20260926 09:31:41.372041 22534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fcae760e36640e38c85f45c6b895378 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:41.371944 22535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fcae760e36640e38c85f45c6b895378 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fcae760e36640e38c85f45c6b895378. Latest consensus state: current_term: 1 leader_uuid: "0fcae760e36640e38c85f45c6b895378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fcae760e36640e38c85f45c6b895378" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34083 } } }
I20260926 09:31:41.372092 22535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fcae760e36640e38c85f45c6b895378 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:41.372231 22542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:41.372779 22542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:41.373548 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 22462
I20260926 09:31:41.373714 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:41.373896 392 external_mini_cluster.cc:1428] Setting key 019128eb828787b9f0870104e360de03
I20260926 09:31:41.374608 22542 catalog_manager.cc:1347] Generated new cluster ID: 9c2e88690f6a40a9a39b5177664d565c
I20260926 09:31:41.374653 22542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:31:41.379699 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:41.374420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54247 (local address 127.0.98.62:34083)
0926 09:31:41.377975 (+ 3555us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:41.377986 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:41.378020 (+ 34us) server_negotiation.cc:407] Connection header received
0926 09:31:41.378149 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:41.378157 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:41.378196 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:41.378327 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:41.379246 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:41.379250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:41.379250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:41.379255 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:41.379298 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:41.379313 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:41.379377 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:41.379453 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 09:31:41.379495 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3497,"thread_start_us":72,"threads_started":1}
I20260926 09:31:41.380640 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:31:41.383365 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:41.382204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54915 (local address 127.0.98.62:34083)
0926 09:31:41.382238 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:41.382240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:41.382354 (+ 114us) server_negotiation.cc:407] Connection header received
0926 09:31:41.382458 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:41.382459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:41.382464 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:41.382492 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:41.383142 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:41.383143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:41.383144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:31:41.383148 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:41.383162 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:31:41.383166 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:41.383195 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:41.383290 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 09:31:41.383303 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 09:31:41.389739 22542 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:41.389943 22542 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:41.397218 22542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fcae760e36640e38c85f45c6b895378: Generated new TSK 0
I20260926 09:31:41.397498 22542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:31:41.405004 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 22462
2026-09-26T09:31:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1314 ms)
[----------] 18 tests from AuthTokenIssuingTest (23835 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:31:41 dist-test-slave-dfzm krb5kdc[22561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:31:41 dist-test-slave-dfzm krb5kdc[22561](info): set up 2 sockets
Sep 26 09:31:41 dist-test-slave-dfzm krb5kdc[22561](info): commencing operation
krb5kdc: starting...
W20260926 09:31:42.453825 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:31:42 dist-test-slave-dfzm krb5kdc[22561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:31:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:31:42.572857 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39327
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39327,127.0.98.61:42775,127.0.98.60:45677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:42.654757 22574 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 09:31:42.654970 22574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:42.654999 22574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:42.655068 22574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:42.656224 22574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:42.656266 22574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:42.656280 22574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:42.656293 22574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:42.656329 22574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:42.657843 22574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39327,127.0.98.61:42775,127.0.98.60:45677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:42.658105 22574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:42.658347 22574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:42.663199 22579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:42.663231 22580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:42.663254 22574 server_base.cc:1034] running on GCE node
W20260926 09:31:42.663216 22582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:42.663686 22574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:42.663993 22574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:42.665165 22574 hybrid_clock.cc:648] HybridClock initialized: now 1790415102665153 us; error 43 us; skew 500 ppm
Sep 26 09:31:42 dist-test-slave-dfzm krb5kdc[22561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:42.667703 22574 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:31:42.668102 22574 webserver.cc:460] Webserver started at http://127.0.98.62:45197/ using document root <none> and password file <none>
I20260926 09:31:42.668315 22574 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:42.668357 22574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:42.668443 22574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:42.669430 22574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "29365cc08d43479abd0f999a8054edc7"
format_stamp: "Formatted at 2026-09-26 09:31:42 on dist-test-slave-dfzm"
server_key: "8cf01f7eaf6f239cfa9efaeaf45e0683"
server_key_iv: "5d17e9b52a33314632a730912f99d978"
server_key_version: "encryptionkey@0"
I20260926 09:31:42.669790 22574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "29365cc08d43479abd0f999a8054edc7"
format_stamp: "Formatted at 2026-09-26 09:31:42 on dist-test-slave-dfzm"
server_key: "8cf01f7eaf6f239cfa9efaeaf45e0683"
server_key_iv: "5d17e9b52a33314632a730912f99d978"
server_key_version: "encryptionkey@0"
I20260926 09:31:42.671000 22574 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:31:42.671707 22589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:42.671857 22574 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:31:42.671939 22574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "29365cc08d43479abd0f999a8054edc7"
format_stamp: "Formatted at 2026-09-26 09:31:42 on dist-test-slave-dfzm"
server_key: "8cf01f7eaf6f239cfa9efaeaf45e0683"
server_key_iv: "5d17e9b52a33314632a730912f99d978"
server_key_version: "encryptionkey@0"
I20260926 09:31:42.671993 22574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:42.704252 22574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:42.706264 22574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:42.706378 22574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:42.710282 22574 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39327
I20260926 09:31:42.710352 22641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39327 every 8 connection(s)
I20260926 09:31:42.710700 22574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:42.711700 22642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:42.713238 22642 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39327 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:31:42.717801 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 22574
I20260926 09:31:42.717934 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:42.718127 392 external_mini_cluster.cc:1428] Setting key a6da3554854509b6d0b4d0c0de742ca9
Sep 26 09:31:42 dist-test-slave-dfzm krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:31:42.723261 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:42.713927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50422 (local address 127.0.98.62:39327)
0926 09:31:42.714137 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:42.714146 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:42.714233 (+ 87us) server_negotiation.cc:407] Connection header received
0926 09:31:42.714797 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:42.714799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:42.714816 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:42.714913 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:42.715456 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:42.716241 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:42.717135 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:42.717433 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:42.717562 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:42.719971 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:42.719979 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:42.719979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:42.719989 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:42.721718 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:42.722092 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:42.722095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:42.722095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:42.722177 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:42.722402 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:42.722403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:42.722403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:42.722628 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:42.722718 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:42.722970 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 09:31:42.723087 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":151,"thread_start_us":61,"threads_started":1}
I20260926 09:31:42.723371 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:42.713908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39327 (local address 127.0.0.1:50422)
0926 09:31:42.714112 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:42.714155 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:42.714632 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:42.714919 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:42.714921 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:42.715199 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:42.715343 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:42.715347 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:42.716356 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:42.716357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:42.717036 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:42.717038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:42.717548 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:42.717549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:42.717648 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:42.718285 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:42.718294 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:42.719882 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:42.721800 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:42.721806 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:42.721808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:42.722003 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:42.722250 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:42.722251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:42.722251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:42.722336 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:42.722723 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:42.722731 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:42.722821 (+ 90us) client_negotiation.cc:769] Sending connection context
0926 09:31:42.722944 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 09:31:42.723185 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":69,"thread_start_us":92,"threads_started":1}
I20260926 09:31:42.724140 22642 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:31:42.724529 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:42.724394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42775 (local address 127.0.0.1:58726)
0926 09:31:42.724450 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:42.724500 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 09:31:42.724694 22592 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42775: connect: Connection refused (error 111)
I20260926 09:31:42.725152 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:42.725043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42775 (local address 127.0.0.1:58734)
0926 09:31:42.725096 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:42.725128 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 09:31:42.725319 22642 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42775: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42775: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 09:31:42.744251 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42775
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39327,127.0.98.61:42775,127.0.98.60:45677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:31:42.774550 22642 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 } attempt: 1
I20260926 09:31:42.775068 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:42.774918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42775 (local address 127.0.0.1:58740)
0926 09:31:42.774982 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:42.775033 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 09:31:42.775408 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:42.775336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42775 (local address 127.0.0.1:58742)
0926 09:31:42.775362 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:42.775389 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260926 09:31:42.775532 22642 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42775: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42775: connect: Connection refused (error 111)
W20260926 09:31:42.825286 22649 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 09:31:42.825496 22649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:42.825513 22649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:42.825564 22649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:42.826701 22649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:42.826745 22649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:42.826759 22649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:42.826771 22649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:42.826826 22649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:42.828282 22649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39327,127.0.98.61:42775,127.0.98.60:45677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:42.828516 22649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:42.828744 22649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:42.833552 22657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:42.833536 22654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:42.833670 22649 server_base.cc:1034] running on GCE node
W20260926 09:31:42.833521 22655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:42.834052 22649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:42.834347 22649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:42.835523 22649 hybrid_clock.cc:648] HybridClock initialized: now 1790415102835496 us; error 50 us; skew 500 ppm
Sep 26 09:31:42 dist-test-slave-dfzm krb5kdc[22561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:42.838871 22649 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 09:31:42.839322 22649 webserver.cc:460] Webserver started at http://127.0.98.61:35019/ using document root <none> and password file <none>
I20260926 09:31:42.839577 22649 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:42.839623 22649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:42.839773 22649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:42.840814 22649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-1/data/instance:
uuid: "60f612f6c6d54fffa2d0382d85cb8a79"
format_stamp: "Formatted at 2026-09-26 09:31:42 on dist-test-slave-dfzm"
server_key: "e3675e386287a76c02be0f4294d054eb"
server_key_iv: "c6753fd9c225daf29222f9679002e94e"
server_key_version: "encryptionkey@0"
I20260926 09:31:42.841176 22649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-1/wal/instance:
uuid: "60f612f6c6d54fffa2d0382d85cb8a79"
format_stamp: "Formatted at 2026-09-26 09:31:42 on dist-test-slave-dfzm"
server_key: "e3675e386287a76c02be0f4294d054eb"
server_key_iv: "c6753fd9c225daf29222f9679002e94e"
server_key_version: "encryptionkey@0"
I20260926 09:31:42.842705 22649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:31:42.843401 22664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:42.843585 22649 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:31:42.843674 22649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-1/wal
uuid: "60f612f6c6d54fffa2d0382d85cb8a79"
format_stamp: "Formatted at 2026-09-26 09:31:42 on dist-test-slave-dfzm"
server_key: "e3675e386287a76c02be0f4294d054eb"
server_key_iv: "c6753fd9c225daf29222f9679002e94e"
server_key_version: "encryptionkey@0"
I20260926 09:31:42.843717 22642 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 } attempt: 2
I20260926 09:31:42.843781 22649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:42.844096 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:42.843976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42775 (local address 127.0.0.1:58752)
0926 09:31:42.844035 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:42.844069 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 09:31:42.844442 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:42.844347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42775 (local address 127.0.0.1:58754)
0926 09:31:42.844390 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:42.844424 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 09:31:42.844534 22642 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42775: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42775: connect: Connection refused (error 111)
I20260926 09:31:42.856763 22649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:42.858665 22649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:42.858829 22649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:42.862851 22649 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42775
I20260926 09:31:42.862882 22716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42775 every 8 connection(s)
I20260926 09:31:42.863250 22649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-1/data/info.pb
I20260926 09:31:42.864193 22717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:42.865774 22717 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39327 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:31:42.869246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 22649
I20260926 09:31:42.869359 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-1/wal/instance
I20260926 09:31:42.869534 392 external_mini_cluster.cc:1428] Setting key c94d741248ad8d4628942568befa7ec1
Sep 26 09:31:42 dist-test-slave-dfzm krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:31:42.875519 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:42.866475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50434 (local address 127.0.98.62:39327)
0926 09:31:42.866532 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:42.866534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:42.866731 (+ 197us) server_negotiation.cc:407] Connection header received
0926 09:31:42.867169 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:42.867170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:42.867181 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:42.867240 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:42.867814 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:42.868283 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:42.869366 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:42.869704 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:42.869771 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:42.872154 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:42.872156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:42.872156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:42.872164 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:42.873951 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:42.874330 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:42.874331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:42.874332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:42.874424 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:42.874788 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:42.874789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:42.874789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:42.874966 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:42.875045 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:42.875323 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 09:31:42.875412 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 09:31:42.875643 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:42.866447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39327 (local address 127.0.0.1:50434)
0926 09:31:42.866604 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:42.866646 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:42.867090 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:42.867309 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:42.867312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:42.867559 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:42.867747 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:42.867750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:42.868379 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:42.868379 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:42.869271 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:42.869274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:42.869829 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:42.869831 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:42.870000 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:42.870502 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:42.870511 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:42.872059 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:42.874032 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:42.874039 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:42.874040 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:42.874252 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:42.874522 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:42.874523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:42.874524 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:42.874714 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:42.875056 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:42.875063 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:42.875185 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 09:31:42.875274 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 09:31:42.875398 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260926 09:31:42.876343 22717 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 26 09:31:42 dist-test-slave-dfzm krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 09:31:42.884589 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:42.876697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58762 (local address 127.0.98.61:42775)
0926 09:31:42.876853 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:42.876857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:42.876907 (+ 50us) server_negotiation.cc:407] Connection header received
0926 09:31:42.876975 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:42.876977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:42.876989 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:42.877054 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:42.877391 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:42.878209 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:42.878935 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:42.879262 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:42.879321 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:42.881469 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:42.881476 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:42.881476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:42.881482 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:42.883264 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:42.883682 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:42.883685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:42.883686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:42.883741 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:42.883946 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:42.883947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:42.883947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:42.884095 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:42.884157 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:42.884379 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 09:31:42.884486 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
I20260926 09:31:42.884598 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:42.876624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42775 (local address 127.0.0.1:58762)
0926 09:31:42.876843 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:42.876855 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:42.876914 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:42.877080 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:42.877081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:42.877246 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:42.877317 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:42.877320 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:42.878287 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:42.878288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:42.878856 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:42.878858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:42.879331 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:42.879332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:42.879387 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:42.879938 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:42.879943 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:42.881371 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:42.883344 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:42.883345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:42.883346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:42.883582 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:42.883813 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:42.883814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:42.883815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:42.883884 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:42.884167 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:42.884168 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:42.884219 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 09:31:42.884298 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 09:31:42.884408 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202}
I20260926 09:31:42.885301 22717 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:31:42.885732 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:42.885567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45677 (local address 127.0.0.1:41372)
0926 09:31:42.885638 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:42.885695 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 09:31:42.885888 22667 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45677: connect: Connection refused (error 111)
I20260926 09:31:42.886341 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:42.886222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45677 (local address 127.0.0.1:41384)
0926 09:31:42.886277 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:42.886310 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 09:31:42.886467 22717 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45677: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45677: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 09:31:42.896904 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45677
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39327,127.0.98.61:42775,127.0.98.60:45677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:31:42.935694 22642 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 } attempt: 3
I20260926 09:31:42.935736 22717 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } attempt: 1
I20260926 09:31:42.936197 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:42.936057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45677 (local address 127.0.0.1:41400)
0926 09:31:42.936117 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:42.936154 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 09:31:42.936566 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:42.936461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45677 (local address 127.0.0.1:41408)
0926 09:31:42.936512 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:42.936539 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 09:31:42.936671 22717 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45677: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45677: connect: Connection refused (error 111)
Sep 26 09:31:42 dist-test-slave-dfzm krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 09:31:42.943557 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:42.936140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58764 (local address 127.0.98.61:42775)
0926 09:31:42.936184 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:42.936186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:42.936196 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:31:42.936226 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:42.936228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:42.936239 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:42.936721 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:42.936759 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:42.937530 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:42.938337 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:42.938695 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:42.938787 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:42.940888 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:42.940889 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:42.940890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:42.940896 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:42.942498 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:42.942874 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:42.942875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:42.942875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:42.942926 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:42.943120 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:42.943120 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:42.943121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:42.943225 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:42.943275 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:42.943432 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 09:31:42.943493 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:31:42.943606 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:42.936059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42775 (local address 127.0.0.1:58764)
0926 09:31:42.936117 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:42.936130 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:42.936173 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:42.936357 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:42.936358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:42.936563 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:42.936645 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:42.936647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:42.937680 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:42.937682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:42.938247 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:42.938250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:42.938798 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:42.938799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:42.938872 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:42.939367 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:42.939372 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:42.940788 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:42.942577 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:42.942578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:42.942579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:42.942797 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:42.942991 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:42.942992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:42.942992 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:42.943059 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:42.943286 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:42.943287 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:42.943338 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 09:31:42.943407 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 09:31:42.943511 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 09:31:42.944139 22642 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:31:42.944463 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:42.944374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45677 (local address 127.0.0.1:41414)
0926 09:31:42.944407 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:42.944438 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 09:31:42.944744 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:42.944688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45677 (local address 127.0.0.1:41428)
0926 09:31:42.944710 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:42.944731 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260926 09:31:42.944842 22642 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45677: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45677: connect: Connection refused (error 111)
I20260926 09:31:42.975996 22642 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } attempt: 1
I20260926 09:31:42.976528 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:42.976372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45677 (local address 127.0.0.1:41434)
0926 09:31:42.976451 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:42.976499 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 09:31:42.976936 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:42.976844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45677 (local address 127.0.0.1:41448)
0926 09:31:42.976892 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:42.976919 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 09:31:42.977046 22642 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45677: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45677: connect: Connection refused (error 111)
W20260926 09:31:42.981343 22724 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 09:31:42.981559 22724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:42.981577 22724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:42.981688 22724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:42.982895 22724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:42.982944 22724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:42.982959 22724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:42.982996 22724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:42.983023 22724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:42.984529 22724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39327,127.0.98.61:42775,127.0.98.60:45677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:42.984794 22724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:42.985049 22724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:42.990079 22732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:42.990079 22730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:42.990126 22729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:42.990325 22724 server_base.cc:1034] running on GCE node
I20260926 09:31:42.990590 22724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:42.990852 22724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:42.992039 22724 hybrid_clock.cc:648] HybridClock initialized: now 1790415102992043 us; error 46 us; skew 500 ppm
Sep 26 09:31:42 dist-test-slave-dfzm krb5kdc[22561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:42.994706 22724 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 09:31:42.995174 22724 webserver.cc:460] Webserver started at http://127.0.98.60:35325/ using document root <none> and password file <none>
I20260926 09:31:42.995407 22724 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:42.995476 22724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:42.995633 22724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:42.996721 22724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-2/data/instance:
uuid: "e63ecb7969164f83b5083a9fa6132a64"
format_stamp: "Formatted at 2026-09-26 09:31:42 on dist-test-slave-dfzm"
server_key: "4ea9dfd2f77de8690624f4375214f2a1"
server_key_iv: "4cb9766d8d949e4a36aeb5a2f3f8f52e"
server_key_version: "encryptionkey@0"
I20260926 09:31:42.997104 22724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-2/wal/instance:
uuid: "e63ecb7969164f83b5083a9fa6132a64"
format_stamp: "Formatted at 2026-09-26 09:31:42 on dist-test-slave-dfzm"
server_key: "4ea9dfd2f77de8690624f4375214f2a1"
server_key_iv: "4cb9766d8d949e4a36aeb5a2f3f8f52e"
server_key_version: "encryptionkey@0"
I20260926 09:31:42.998571 22724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:31:42.999343 22739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:42.999519 22724 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:31:42.999620 22724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-2/wal
uuid: "e63ecb7969164f83b5083a9fa6132a64"
format_stamp: "Formatted at 2026-09-26 09:31:42 on dist-test-slave-dfzm"
server_key: "4ea9dfd2f77de8690624f4375214f2a1"
server_key_iv: "4cb9766d8d949e4a36aeb5a2f3f8f52e"
server_key_version: "encryptionkey@0"
I20260926 09:31:42.999706 22724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:43.004801 22717 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } attempt: 2
I20260926 09:31:43.005231 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.005105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45677 (local address 127.0.0.1:41450)
0926 09:31:43.005165 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:43.005204 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 09:31:43.005662 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.005522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45677 (local address 127.0.0.1:41464)
0926 09:31:43.005574 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:43.005635 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 09:31:43.005795 22717 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45677: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45677: connect: Connection refused (error 111)
I20260926 09:31:43.009546 22724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:43.011898 22724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:43.012066 22724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:43.016243 22724 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45677
I20260926 09:31:43.016350 22791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45677 every 8 connection(s)
I20260926 09:31:43.016659 22724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-2/data/info.pb
I20260926 09:31:43.017769 22792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:43.019295 22792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39327 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:31:43.021739 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 22724
I20260926 09:31:43.021849 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414839069479-392-0/minicluster-data/master-2/wal/instance
I20260926 09:31:43.022022 392 external_mini_cluster.cc:1428] Setting key 6483f5f8dd57c2432c0ede1d783ed88b
Sep 26 09:31:43 dist-test-slave-dfzm krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 09:31:43 dist-test-slave-dfzm krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 09:31:43.029058 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.020064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50442 (local address 127.0.98.62:39327)
0926 09:31:43.020128 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.020130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.020306 (+ 176us) server_negotiation.cc:407] Connection header received
0926 09:31:43.020744 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.020747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.020762 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.020838 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:43.021478 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.022107 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.023357 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.023623 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.023718 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.025949 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:43.025951 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:43.025951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:43.025959 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:43.027497 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:43.027958 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:43.027960 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:43.027960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:43.028023 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:43.028338 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:43.028339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:43.028339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:43.028465 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:43.028570 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:43.028870 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 09:31:43.028957 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 09:31:43.029304 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.019978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39327 (local address 127.0.0.1:50442)
0926 09:31:43.020141 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:43.020183 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:43.020649 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:43.020944 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:43.020948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:43.021197 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:43.021402 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:43.021406 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.022212 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.022213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:43.023265 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:43.023267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.023748 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.023750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:43.023871 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.024448 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:43.024456 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:43.025856 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:43.027582 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:43.027588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:43.027590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:43.027882 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:43.028095 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:43.028095 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:43.028095 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:43.028224 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:43.028562 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:43.028569 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:43.028722 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 09:31:43.028841 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 09:31:43.029031 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260926 09:31:43.030205 22792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:31:43.031332 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.022449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41476 (local address 127.0.98.60:45677)
0926 09:31:43.022605 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.022609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.022676 (+ 67us) server_negotiation.cc:407] Connection header received
0926 09:31:43.022736 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.022738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.022758 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.022834 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:43.023491 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.024272 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.025034 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.025348 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.025439 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.028236 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:43.028242 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:43.028242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:43.028249 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:43.030072 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:43.030466 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:43.030469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:43.030469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:43.030540 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:43.030753 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:43.030754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:43.030754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:43.030904 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:43.031069 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:43.031139 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 09:31:43.031219 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
Sep 26 09:31:43 dist-test-slave-dfzm krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 09:31:43 dist-test-slave-dfzm krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:31:43.037945 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.030546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58780 (local address 127.0.98.61:42775)
0926 09:31:43.030590 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.030592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.030632 (+ 40us) server_negotiation.cc:407] Connection header received
0926 09:31:43.030706 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.030707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.030716 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.030764 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:43.031014 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.031624 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.032469 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.032699 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.032777 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.034775 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:43.034777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:43.034777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:43.034786 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:43.036762 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:43.037121 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:43.037121 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:43.037122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:43.037169 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:43.037383 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:43.037384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:43.037384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:43.037503 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:43.037583 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:43.037776 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 09:31:43.037851 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:31:43.037976 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.030498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42775 (local address 127.0.0.1:58780)
0926 09:31:43.030592 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:43.030603 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:43.030647 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:43.030765 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:43.030766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:43.030900 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:43.030934 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:43.030936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.031713 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.031714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:43.032379 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:43.032381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.032772 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.032774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:43.032833 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.033288 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:43.033293 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:43.034692 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:43.036840 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:43.036841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:43.036841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:43.037045 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:43.037236 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:43.037237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:43.037237 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:43.037299 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:43.037577 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:43.037578 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:43.037666 (+ 88us) client_negotiation.cc:769] Sending connection context
0926 09:31:43.037751 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 09:31:43.037881 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 09:31:43.038600 22792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:31:43.040589 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.032456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.0.98.62:39327)
0926 09:31:43.032515 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.032517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.032553 (+ 36us) server_negotiation.cc:407] Connection header received
0926 09:31:43.032620 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.032621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.032631 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.032686 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:43.033208 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.033952 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.034886 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.035071 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.035123 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.037793 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:43.037795 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:43.037795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:43.037802 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:43.039466 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:43.039810 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:43.039811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:43.039811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:43.039865 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:43.040084 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:43.040085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:43.040086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:43.040195 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:43.040366 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:43.040403 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 09:31:43.040480 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 09:31:43 dist-test-slave-dfzm krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 09:31:43.045567 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.038875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45677 (local address 127.0.0.1:41490)
0926 09:31:43.038924 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:43.038933 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:43.038971 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:43.039082 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:43.039082 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:43.039240 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:43.039276 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:43.039278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.039968 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.039969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:43.040437 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:43.040438 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.040905 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.040906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:43.040952 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.041344 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:43.041349 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:43.042759 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:43.044578 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:43.044579 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:43.044579 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:43.044777 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:43.045019 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:43.045019 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:43.045020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:43.045077 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:43.045343 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:43.045343 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:43.045376 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 09:31:43.045429 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 09:31:43.045505 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 09:31:43.045753 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.038912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41490 (local address 127.0.98.60:45677)
0926 09:31:43.038953 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.038955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.038963 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:31:43.039024 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.039025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.039034 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.039081 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:43.039374 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.039884 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.040525 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.040848 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.040909 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.042843 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:43.042844 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:43.042844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:43.042852 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:43.044493 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:43.044856 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:43.044858 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:43.044858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:43.044927 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:43.045144 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:43.045145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:43.045146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:43.045272 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:43.045348 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:43.045461 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 09:31:43.045558 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":28}
Sep 26 09:31:43 dist-test-slave-dfzm krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 09:31:43.047530 22792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e63ecb7969164f83b5083a9fa6132a64: Bootstrap starting.
I20260926 09:31:43.048517 22792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e63ecb7969164f83b5083a9fa6132a64: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:43.048871 22792 log.cc:824] T 00000000000000000000000000000000 P e63ecb7969164f83b5083a9fa6132a64: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:43.049096 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.041353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58782 (local address 127.0.98.61:42775)
0926 09:31:43.041392 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.041394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.041399 (+ 5us) server_negotiation.cc:407] Connection header received
0926 09:31:43.041415 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.041415 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.041427 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.041465 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:43.042076 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.042816 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.043454 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.043749 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.043834 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.046525 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:43.046527 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:43.046527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:43.046534 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:43.047943 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:43.048339 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:43.048340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:43.048340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:43.048386 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:43.048600 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:43.048600 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:43.048601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:43.048710 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:43.048815 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:43.048925 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 09:31:43.049033 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:31:43.049986 22792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e63ecb7969164f83b5083a9fa6132a64: No bootstrap required, opened a new log
I20260926 09:31:43.050513 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:31:43.051913 22792 raft_consensus.cc:348] T 00000000000000000000000000000000 P e63ecb7969164f83b5083a9fa6132a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29365cc08d43479abd0f999a8054edc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39327 } } peers { permanent_uuid: "60f612f6c6d54fffa2d0382d85cb8a79" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 } } peers { permanent_uuid: "e63ecb7969164f83b5083a9fa6132a64" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } }
I20260926 09:31:43.052047 22792 raft_consensus.cc:729] T 00000000000000000000000000000000 P e63ecb7969164f83b5083a9fa6132a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e63ecb7969164f83b5083a9fa6132a64, State: Initialized, Role: FOLLOWER
I20260926 09:31:43.052132 22792 consensus_queue.cc:260] T 00000000000000000000000000000000 P e63ecb7969164f83b5083a9fa6132a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29365cc08d43479abd0f999a8054edc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39327 } } peers { permanent_uuid: "60f612f6c6d54fffa2d0382d85cb8a79" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 } } peers { permanent_uuid: "e63ecb7969164f83b5083a9fa6132a64" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } }
I20260926 09:31:43.052173 22642 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } attempt: 2
I20260926 09:31:43.051545 22616 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "29365cc08d43479abd0f999a8054edc7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50450
I20260926 09:31:43.052444 22792 sys_catalog.cc:564] T 00000000000000000000000000000000 P e63ecb7969164f83b5083a9fa6132a64 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:43.053910 22798 sys_catalog.cc:455] T 00000000000000000000000000000000 P e63ecb7969164f83b5083a9fa6132a64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29365cc08d43479abd0f999a8054edc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39327 } } peers { permanent_uuid: "60f612f6c6d54fffa2d0382d85cb8a79" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 } } peers { permanent_uuid: "e63ecb7969164f83b5083a9fa6132a64" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } } }
I20260926 09:31:43.054131 22798 sys_catalog.cc:458] T 00000000000000000000000000000000 P e63ecb7969164f83b5083a9fa6132a64 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 09:31:43.054421 22809 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e63ecb7969164f83b5083a9fa6132a64: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:31:43.054466 22809 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 26 09:31:43 dist-test-slave-dfzm krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 09:31:43.054697 22724 master_runner.cc:186] Error getting master registration for 127.0.98.62:39327: OK, instance_id { permanent_uuid: "29365cc08d43479abd0f999a8054edc7" instance_seqno: 1790415102709092 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 09:31:43.057554 22724 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 09:31:43.059504 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.052515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41506 (local address 127.0.98.60:45677)
0926 09:31:43.052609 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.052611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.052622 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:31:43.052645 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.052646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.052659 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.052730 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:43.053122 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.053953 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.054857 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.055063 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.055137 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.056849 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:43.056851 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:43.056851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:43.056858 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:43.058328 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:43.058695 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:43.058696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:43.058696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:43.058747 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:43.058976 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:43.058977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:43.058978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:43.059110 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:43.059184 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:43.059344 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 09:31:43.059415 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":73}
I20260926 09:31:43.059511 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.052394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45677 (local address 127.0.0.1:41506)
0926 09:31:43.052441 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:43.052452 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:43.052506 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:43.052762 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:43.052763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:43.052961 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:43.053030 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:43.053032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.054060 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.054061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:43.054702 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:43.054705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.055135 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.055136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:43.055190 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.055628 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:43.055633 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:43.056763 (+ 1130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:43.058437 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:43.058438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:43.058438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:43.058610 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:43.058822 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:43.058823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:43.058824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:43.058881 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:43.059189 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:43.059190 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:43.059234 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 09:31:43.059320 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 09:31:43.059423 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 09:31:43.061070 22642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7: Bootstrap starting.
I20260926 09:31:43.061843 22642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:43.062151 22642 log.cc:824] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:43.063021 22642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7: No bootstrap required, opened a new log
I20260926 09:31:43.064416 22642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29365cc08d43479abd0f999a8054edc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39327 } } peers { permanent_uuid: "60f612f6c6d54fffa2d0382d85cb8a79" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 } } peers { permanent_uuid: "e63ecb7969164f83b5083a9fa6132a64" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } }
I20260926 09:31:43.064575 22642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29365cc08d43479abd0f999a8054edc7, State: Initialized, Role: FOLLOWER
I20260926 09:31:43.064667 22642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29365cc08d43479abd0f999a8054edc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39327 } } peers { permanent_uuid: "60f612f6c6d54fffa2d0382d85cb8a79" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 } } peers { permanent_uuid: "e63ecb7969164f83b5083a9fa6132a64" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } }
I20260926 09:31:43.064987 22642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:43.064968 22812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29365cc08d43479abd0f999a8054edc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39327 } } peers { permanent_uuid: "60f612f6c6d54fffa2d0382d85cb8a79" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 } } peers { permanent_uuid: "e63ecb7969164f83b5083a9fa6132a64" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } } }
I20260926 09:31:43.065027 22616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 09:31:43.065079 22812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:31:43.065115 22616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:43.065887 22616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29365cc08d43479abd0f999a8054edc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39327 } } peers { permanent_uuid: "60f612f6c6d54fffa2d0382d85cb8a79" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 } } peers { permanent_uuid: "e63ecb7969164f83b5083a9fa6132a64" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } }
I20260926 09:31:43.066157 22616 leader_election.cc:288] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7 [CANDIDATE]: Term 1 election: Requested vote from peers 60f612f6c6d54fffa2d0382d85cb8a79 (127.0.98.61:42775), e63ecb7969164f83b5083a9fa6132a64 (127.0.98.60:45677)
I20260926 09:31:43.066577 22766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29365cc08d43479abd0f999a8054edc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e63ecb7969164f83b5083a9fa6132a64"
I20260926 09:31:43.066700 22766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e63ecb7969164f83b5083a9fa6132a64 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:43.067711 22766 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e63ecb7969164f83b5083a9fa6132a64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29365cc08d43479abd0f999a8054edc7 in term 1.
I20260926 09:31:43.068060 22592 leader_election.cc:302] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29365cc08d43479abd0f999a8054edc7, e63ecb7969164f83b5083a9fa6132a64; no voters:
I20260926 09:31:43.068249 22812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:43.068389 22812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7 [term 1 LEADER]: Becoming Leader. State: Replica: 29365cc08d43479abd0f999a8054edc7, State: Running, Role: LEADER
I20260926 09:31:43.068475 22812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29365cc08d43479abd0f999a8054edc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39327 } } peers { permanent_uuid: "60f612f6c6d54fffa2d0382d85cb8a79" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 } } peers { permanent_uuid: "e63ecb7969164f83b5083a9fa6132a64" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } }
I20260926 09:31:43.068881 22812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29365cc08d43479abd0f999a8054edc7. Latest consensus state: current_term: 1 leader_uuid: "29365cc08d43479abd0f999a8054edc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29365cc08d43479abd0f999a8054edc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39327 } } peers { permanent_uuid: "60f612f6c6d54fffa2d0382d85cb8a79" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 } } peers { permanent_uuid: "e63ecb7969164f83b5083a9fa6132a64" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } } }
I20260926 09:31:43.068969 22812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7 [sys.catalog]: This master's current role is: LEADER
W20260926 09:31:43.069651 22823 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:31:43.069706 22823 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:31:43.069478 22574 master_runner.cc:186] Error getting master registration for 127.0.98.61:42775: OK, instance_id { permanent_uuid: "60f612f6c6d54fffa2d0382d85cb8a79" instance_seqno: 1790415102861715 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 09:31:43.072002 22574 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 09:31:43.073964 22833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:43.074182 22833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:43.076138 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.067338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50464 (local address 127.0.98.62:39327)
0926 09:31:43.067380 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.067382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.068045 (+ 663us) server_negotiation.cc:407] Connection header received
0926 09:31:43.068165 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.068165 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.068177 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.068235 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:43.069101 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.069838 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.070528 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.070811 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.070873 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.072947 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:43.072948 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:43.072948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:43.072956 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:43.074675 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:43.075045 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:43.075046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:43.075047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:43.075111 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:43.075311 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:43.075313 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:43.075313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:43.075441 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:43.075562 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:43.075975 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 09:31:43.076047 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:31:43.082633 22766 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e63ecb7969164f83b5083a9fa6132a64 [term 1 FOLLOWER]: Refusing update from remote peer 29365cc08d43479abd0f999a8054edc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:31:43.082991 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.067427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.0.98.61:42775)
0926 09:31:43.067470 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.067472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.074038 (+ 6566us) server_negotiation.cc:407] Connection header received
0926 09:31:43.074067 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.074069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.074081 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.074136 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:43.075508 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.076222 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.077209 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.077522 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.077562 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.079132 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:43.079133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:43.079133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:43.079142 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:43.081070 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:43.081486 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:43.081488 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:43.081488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:43.081546 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:43.081764 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:43.081765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:43.081765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:43.081897 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:43.082269 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:43.082772 (+ 503us) server_negotiation.cc:299] Negotiation successful
0926 09:31:43.082860 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:31:43.083374 22831 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e63ecb7969164f83b5083a9fa6132a64" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:31:43.085021 22798 sys_catalog.cc:455] T 00000000000000000000000000000000 P e63ecb7969164f83b5083a9fa6132a64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29365cc08d43479abd0f999a8054edc7. Latest consensus state: current_term: 1 leader_uuid: "29365cc08d43479abd0f999a8054edc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29365cc08d43479abd0f999a8054edc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39327 } } peers { permanent_uuid: "60f612f6c6d54fffa2d0382d85cb8a79" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 } } peers { permanent_uuid: "e63ecb7969164f83b5083a9fa6132a64" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } } }
I20260926 09:31:43.085119 22798 sys_catalog.cc:458] T 00000000000000000000000000000000 P e63ecb7969164f83b5083a9fa6132a64 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:31:43.085606 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.067758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41514 (local address 127.0.98.60:45677)
0926 09:31:43.067795 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.067797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.075918 (+ 8121us) server_negotiation.cc:407] Connection header received
0926 09:31:43.076296 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.076297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.076308 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.076337 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:43.077005 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.077716 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.079228 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.079555 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.079597 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.081707 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:43.081708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:43.081709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:43.081715 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:43.083467 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:43.084344 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:43.084345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:43.084345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:43.084411 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:43.084721 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:43.084722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:43.084723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:43.084860 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:43.084933 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:43.085399 (+ 466us) server_negotiation.cc:299] Negotiation successful
0926 09:31:43.085486 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:31:43.086580 22831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29365cc08d43479abd0f999a8054edc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29365cc08d43479abd0f999a8054edc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39327 } } peers { permanent_uuid: "60f612f6c6d54fffa2d0382d85cb8a79" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 } } peers { permanent_uuid: "e63ecb7969164f83b5083a9fa6132a64" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } } }
I20260926 09:31:43.086687 22831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:43.087100 22798 sys_catalog.cc:455] T 00000000000000000000000000000000 P e63ecb7969164f83b5083a9fa6132a64 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29365cc08d43479abd0f999a8054edc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29365cc08d43479abd0f999a8054edc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39327 } } peers { permanent_uuid: "60f612f6c6d54fffa2d0382d85cb8a79" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 } } peers { permanent_uuid: "e63ecb7969164f83b5083a9fa6132a64" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } } }
I20260926 09:31:43.087180 22798 sys_catalog.cc:458] T 00000000000000000000000000000000 P e63ecb7969164f83b5083a9fa6132a64 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:31:43.087368 22833 catalog_manager.cc:1347] Generated new cluster ID: 5cc48acaf57243b4aa27fbc133bf4055
I20260926 09:31:43.087417 22833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:31:43.093820 22833 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:43.094233 22836 mvcc.cc:205] Tried to move back new op lower bound from 7333540262224007168 to 7333540262169743360. Current Snapshot: MvccSnapshot[applied={T|T < 7333540262224007168}]
I20260926 09:31:43.095304 22833 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:43.096949 22717 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } attempt: 3
Sep 26 09:31:43 dist-test-slave-dfzm krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 09:31:43.104879 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.097179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45677 (local address 127.0.0.1:41530)
0926 09:31:43.097256 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:43.097268 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:43.097306 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:43.097496 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:43.097497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:43.097720 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:43.097765 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:43.097766 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.098634 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.098635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:43.099449 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:43.099451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.099857 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.099858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:43.099910 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.100355 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:43.100369 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:43.101876 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:43.103842 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:43.103844 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:43.103844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:43.104065 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:43.104266 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:43.104267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:43.104267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:43.104320 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:43.104552 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:43.104553 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:43.104601 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 09:31:43.104665 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 09:31:43.104766 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 09:31:43.105253 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.097299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41530 (local address 127.0.98.60:45677)
0926 09:31:43.097376 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.097378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.097386 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:31:43.097416 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.097417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.097426 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.097470 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:43.097824 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.098544 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.099523 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.099797 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.099859 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.101965 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:43.101966 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:43.101966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:43.101973 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:43.103765 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:43.104145 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:43.104146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:43.104147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:43.104201 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:43.104381 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:43.104381 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:43.104382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:43.104499 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:43.104561 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:43.105085 (+ 524us) server_negotiation.cc:299] Negotiation successful
0926 09:31:43.105159 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 09:31:43.107436 22717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f612f6c6d54fffa2d0382d85cb8a79: Bootstrap starting.
I20260926 09:31:43.108168 22717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60f612f6c6d54fffa2d0382d85cb8a79: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:43.108475 22717 log.cc:824] T 00000000000000000000000000000000 P 60f612f6c6d54fffa2d0382d85cb8a79: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:43.109375 22717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f612f6c6d54fffa2d0382d85cb8a79: No bootstrap required, opened a new log
I20260926 09:31:43.112318 22717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60f612f6c6d54fffa2d0382d85cb8a79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29365cc08d43479abd0f999a8054edc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39327 } } peers { permanent_uuid: "60f612f6c6d54fffa2d0382d85cb8a79" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 } } peers { permanent_uuid: "e63ecb7969164f83b5083a9fa6132a64" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } }
I20260926 09:31:43.112625 22717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60f612f6c6d54fffa2d0382d85cb8a79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60f612f6c6d54fffa2d0382d85cb8a79, State: Initialized, Role: FOLLOWER
I20260926 09:31:43.112813 22717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60f612f6c6d54fffa2d0382d85cb8a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29365cc08d43479abd0f999a8054edc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39327 } } peers { permanent_uuid: "60f612f6c6d54fffa2d0382d85cb8a79" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 } } peers { permanent_uuid: "e63ecb7969164f83b5083a9fa6132a64" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } }
I20260926 09:31:43.113451 22691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29365cc08d43479abd0f999a8054edc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "60f612f6c6d54fffa2d0382d85cb8a79"
I20260926 09:31:43.113505 22690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60f612f6c6d54fffa2d0382d85cb8a79 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:43.114655 22690 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60f612f6c6d54fffa2d0382d85cb8a79 [term 1 FOLLOWER]: Refusing update from remote peer 29365cc08d43479abd0f999a8054edc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:31:43.114985 22831 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60f612f6c6d54fffa2d0382d85cb8a79" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:31:43.116321 22691 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60f612f6c6d54fffa2d0382d85cb8a79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29365cc08d43479abd0f999a8054edc7 in term 1.
I20260926 09:31:43.113408 22717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60f612f6c6d54fffa2d0382d85cb8a79 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:43.118295 22843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f612f6c6d54fffa2d0382d85cb8a79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29365cc08d43479abd0f999a8054edc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29365cc08d43479abd0f999a8054edc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39327 } } peers { permanent_uuid: "60f612f6c6d54fffa2d0382d85cb8a79" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 } } peers { permanent_uuid: "e63ecb7969164f83b5083a9fa6132a64" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } } }
I20260926 09:31:43.118458 22843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f612f6c6d54fffa2d0382d85cb8a79 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:31:43.118554 22843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f612f6c6d54fffa2d0382d85cb8a79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29365cc08d43479abd0f999a8054edc7. Latest consensus state: current_term: 1 leader_uuid: "29365cc08d43479abd0f999a8054edc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29365cc08d43479abd0f999a8054edc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39327 } } peers { permanent_uuid: "60f612f6c6d54fffa2d0382d85cb8a79" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 } } peers { permanent_uuid: "e63ecb7969164f83b5083a9fa6132a64" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } } }
I20260926 09:31:43.118678 22843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f612f6c6d54fffa2d0382d85cb8a79 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:31:43.118762 22843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f612f6c6d54fffa2d0382d85cb8a79 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29365cc08d43479abd0f999a8054edc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29365cc08d43479abd0f999a8054edc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39327 } } peers { permanent_uuid: "60f612f6c6d54fffa2d0382d85cb8a79" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 } } peers { permanent_uuid: "e63ecb7969164f83b5083a9fa6132a64" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } } }
I20260926 09:31:43.118886 22843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f612f6c6d54fffa2d0382d85cb8a79 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:31:43.119525 22833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7: Generated new TSK 0
I20260926 09:31:43.119585 22812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29365cc08d43479abd0f999a8054edc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29365cc08d43479abd0f999a8054edc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39327 } } peers { permanent_uuid: "60f612f6c6d54fffa2d0382d85cb8a79" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42775 } } peers { permanent_uuid: "e63ecb7969164f83b5083a9fa6132a64" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45677 } } }
I20260926 09:31:43.119661 22812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29365cc08d43479abd0f999a8054edc7 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:43.119724 22833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:31:43.121786 22859 catalog_manager.cc:1259] Loaded cluster ID: 5cc48acaf57243b4aa27fbc133bf4055
I20260926 09:31:43.121892 22859 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 60f612f6c6d54fffa2d0382d85cb8a79: loading cluster ID for follower catalog manager: success
I20260926 09:31:43.123138 22859 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 60f612f6c6d54fffa2d0382d85cb8a79: acquiring CA information for follower catalog manager: success
I20260926 09:31:43.123632 22859 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 60f612f6c6d54fffa2d0382d85cb8a79: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 09:31:43.143615 22645 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:43.141885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50478 (local address 127.0.98.62:39327)
0926 09:31:43.141934 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.141936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.142919 (+ 983us) server_negotiation.cc:407] Connection header received
0926 09:31:43.143033 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.143035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.143046 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.143078 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:43.143540 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50478: BlockingRecv error: recv got EOF from 127.0.0.1:50478 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 09:31:43.144533 22721 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:43.142343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58808 (local address 127.0.98.61:42775)
0926 09:31:43.142385 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.142386 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.143956 (+ 1570us) server_negotiation.cc:407] Connection header received
0926 09:31:43.144056 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.144058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.144069 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.144097 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:43.144462 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58808: BlockingRecv error: recv got EOF from 127.0.0.1:58808 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 09:31:43.145344 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:43.142647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41540 (local address 127.0.98.60:45677)
0926 09:31:43.142684 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.142686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.144844 (+ 2158us) server_negotiation.cc:407] Connection header received
0926 09:31:43.144939 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.144940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.144951 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.144980 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:43.145272 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41540: BlockingRecv error: recv got EOF from 127.0.0.1:41540 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:31:43.150779 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.147106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58818 (local address 127.0.98.61:42775)
0926 09:31:43.147143 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.147145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.147352 (+ 207us) server_negotiation.cc:407] Connection header received
0926 09:31:43.147452 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.147453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.147495 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.147539 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:43.147783 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.148571 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.149807 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.150122 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.150162 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.150402 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:43.150530 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:43.150578 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:43.150688 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 09:31:43.150706 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 09:31:43.154263 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:43.153371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41542 (local address 127.0.98.60:45677)
0926 09:31:43.153410 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.153412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.153724 (+ 312us) server_negotiation.cc:407] Connection header received
0926 09:31:43.153833 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.153834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.153853 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.153883 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:43.154211 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41542: BlockingRecv error: recv got EOF from 127.0.0.1:41542 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:31:43.159780 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.156505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58824 (local address 127.0.98.61:42775)
0926 09:31:43.156540 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.156542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.156773 (+ 231us) server_negotiation.cc:407] Connection header received
0926 09:31:43.156864 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.156865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.156875 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.156901 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:43.157062 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.157847 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.158957 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.159269 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.159321 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.159524 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:43.159557 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:43.159585 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:43.159690 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 09:31:43.159707 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 09:31:43.163055 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:43.162288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41550 (local address 127.0.98.60:45677)
0926 09:31:43.162338 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.162340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.162545 (+ 205us) server_negotiation.cc:407] Connection header received
0926 09:31:43.162637 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.162638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.162648 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.162676 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:43.163001 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41550: BlockingRecv error: recv got EOF from 127.0.0.1:41550 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 09:31:43.169528 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.166229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58828 (local address 127.0.98.61:42775)
0926 09:31:43.166263 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.166265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.166603 (+ 338us) server_negotiation.cc:407] Connection header received
0926 09:31:43.166696 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.166697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.166706 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.166733 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:43.166893 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.167650 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.168744 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.169038 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.169077 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.169261 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:43.169293 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:43.169332 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:43.169438 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 09:31:43.169456 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 09:31:43.172693 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:43.171896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41552 (local address 127.0.98.60:45677)
0926 09:31:43.171935 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.171937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.172137 (+ 200us) server_negotiation.cc:407] Connection header received
0926 09:31:43.172228 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.172230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.172240 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.172279 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:43.172642 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41552: BlockingRecv error: recv got EOF from 127.0.0.1:41552 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 09:31:43.185170 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.181873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58836 (local address 127.0.98.61:42775)
0926 09:31:43.181914 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.181916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.182167 (+ 251us) server_negotiation.cc:407] Connection header received
0926 09:31:43.182267 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.182268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.182278 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.182345 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:43.182520 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.183259 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.184386 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.184687 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.184724 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.184912 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:43.184945 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:43.184971 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:43.185072 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 09:31:43.185089 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 09:31:43.188409 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:43.187598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41562 (local address 127.0.98.60:45677)
0926 09:31:43.187636 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.187638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.187864 (+ 226us) server_negotiation.cc:407] Connection header received
0926 09:31:43.187960 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.187961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.187972 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.188000 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:43.188351 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41562: BlockingRecv error: recv got EOF from 127.0.0.1:41562 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:31:43.201207 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.197841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58852 (local address 127.0.98.61:42775)
0926 09:31:43.197882 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.197884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.198163 (+ 279us) server_negotiation.cc:407] Connection header received
0926 09:31:43.198279 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.198281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.198291 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.198320 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:43.198495 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.199259 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.200391 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.200693 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.200733 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.200929 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:43.200963 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:43.200996 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:43.201099 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 09:31:43.201117 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 09:31:43.204527 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:43.203680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41572 (local address 127.0.98.60:45677)
0926 09:31:43.203729 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.203731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.203980 (+ 249us) server_negotiation.cc:407] Connection header received
0926 09:31:43.204072 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.204073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.204084 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.204113 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:43.204470 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41572: BlockingRecv error: recv got EOF from 127.0.0.1:41572 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:31:43.225567 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.221970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58860 (local address 127.0.98.61:42775)
0926 09:31:43.222015 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.222018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.222344 (+ 326us) server_negotiation.cc:407] Connection header received
0926 09:31:43.222461 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.222462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.222473 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.222505 (+ 32us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:43.222767 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.223515 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.224723 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.224995 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.225043 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.225286 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:43.225323 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:43.225352 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:43.225464 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 09:31:43.225485 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 09:31:43.229218 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:43.228290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41586 (local address 127.0.98.60:45677)
0926 09:31:43.228333 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.228335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.228594 (+ 259us) server_negotiation.cc:407] Connection header received
0926 09:31:43.228720 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.228721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.228733 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.228765 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:43.229145 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41586: BlockingRecv error: recv got EOF from 127.0.0.1:41586 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:31:43.266104 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.262730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58864 (local address 127.0.98.61:42775)
0926 09:31:43.262777 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.262779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.263091 (+ 312us) server_negotiation.cc:407] Connection header received
0926 09:31:43.263148 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.263150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.263161 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.263237 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:43.263380 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.264099 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.265251 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.265605 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.265681 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.265828 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:43.265863 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:43.265917 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:43.265987 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 09:31:43.266021 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 09:31:43.268937 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:43.268156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41588 (local address 127.0.98.60:45677)
0926 09:31:43.268235 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.268237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.268409 (+ 172us) server_negotiation.cc:407] Connection header received
0926 09:31:43.268443 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.268444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.268456 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.268524 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:43.268878 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41588: BlockingRecv error: recv got EOF from 127.0.0.1:41588 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 09:31:43.336290 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.334006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58866 (local address 127.0.98.61:42775)
0926 09:31:43.334084 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.334087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.334224 (+ 137us) server_negotiation.cc:407] Connection header received
0926 09:31:43.334271 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.334272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.334282 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.334326 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:43.334442 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.334935 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.335652 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.335853 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.335920 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.336039 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:43.336067 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:43.336107 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:43.336156 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 09:31:43.336173 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 09:31:43.338725 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:43.338246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41600 (local address 127.0.98.60:45677)
0926 09:31:43.338291 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.338293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.338313 (+ 20us) server_negotiation.cc:407] Connection header received
0926 09:31:43.338364 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.338365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.338375 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.338416 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:43.338677 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41600: BlockingRecv error: recv got EOF from 127.0.0.1:41600 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:31:43.470198 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.467791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58870 (local address 127.0.98.61:42775)
0926 09:31:43.467859 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.467862 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.468018 (+ 156us) server_negotiation.cc:407] Connection header received
0926 09:31:43.468063 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.468064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.468073 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.468127 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:43.468257 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.468718 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.469509 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.469777 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.469835 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.469972 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:43.469998 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:43.470036 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:43.470084 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 09:31:43.470099 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 09:31:43.472486 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:43.471833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41616 (local address 127.0.98.60:45677)
0926 09:31:43.471916 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.471919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.471973 (+ 54us) server_negotiation.cc:407] Connection header received
0926 09:31:43.472056 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.472057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.472067 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.472113 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:43.472427 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41616: BlockingRecv error: recv got EOF from 127.0.0.1:41616 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 09:31:43.731927 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:43.729710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.0.98.61:42775)
0926 09:31:43.729778 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.729780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.729789 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:31:43.729817 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.729818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.729827 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.729880 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:43.730093 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.730569 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.731323 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:43.731505 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:43.731562 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:43.731685 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:43.731710 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:43.731749 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:43.731799 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 09:31:43.731814 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 09:31:43.734328 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:43.733656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41620 (local address 127.0.98.60:45677)
0926 09:31:43.733728 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:43.733731 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:43.733741 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:31:43.733805 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:43.733806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:43.733825 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:43.733892 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:43.734250 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41620: BlockingRecv error: recv got EOF from 127.0.0.1:41620 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 09:31:44.054886 22809 catalog_manager.cc:1259] Loaded cluster ID: 5cc48acaf57243b4aa27fbc133bf4055
I20260926 09:31:44.054986 22809 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e63ecb7969164f83b5083a9fa6132a64: loading cluster ID for follower catalog manager: success
I20260926 09:31:44.055783 22809 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e63ecb7969164f83b5083a9fa6132a64: acquiring CA information for follower catalog manager: success
I20260926 09:31:44.056068 22809 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e63ecb7969164f83b5083a9fa6132a64: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 09:31:44.250093 22976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:44.247476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58888 (local address 127.0.98.61:42775)
0926 09:31:44.247593 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:44.247595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:44.247672 (+ 77us) server_negotiation.cc:407] Connection header received
0926 09:31:44.247765 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:44.247767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:44.247779 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:44.247856 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:44.248018 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:44.248537 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:44.249321 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:44.249533 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:44.249644 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:44.249795 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:44.249834 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:44.249900 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:44.249963 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 09:31:44.249985 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":60,"threads_started":1}
I20260926 09:31:44.254269 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:44.251761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41636 (local address 127.0.98.60:45677)
0926 09:31:44.251907 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:44.251909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:44.251958 (+ 49us) server_negotiation.cc:407] Connection header received
0926 09:31:44.252016 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:44.252016 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:44.252077 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:44.252153 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:44.252238 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:44.252706 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:44.253491 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:44.253737 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:44.253816 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:44.253919 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:44.254036 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:44.254123 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:44.254149 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 09:31:44.254169 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":49,"threads_started":1}
I20260926 09:31:44.258903 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:44.255950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50494 (local address 127.0.98.62:39327)
0926 09:31:44.256063 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:44.256065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:44.256075 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:31:44.256198 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:44.256200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:44.256253 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:44.256381 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:44.256474 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:44.257080 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:44.257922 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:44.258213 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:44.258316 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:44.258537 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:44.258647 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:44.258716 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:44.258774 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 09:31:44.258798 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":55,"threads_started":1}
I20260926 09:31:44.259292 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:44.256176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41640 (local address 127.0.98.60:45677)
0926 09:31:44.256219 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:44.256221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:44.256462 (+ 241us) server_negotiation.cc:407] Connection header received
0926 09:31:44.256569 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:44.256570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:44.256581 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:44.256631 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:44.256753 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:44.257424 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:44.258506 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:44.258790 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:44.258829 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:44.259039 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:44.259077 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:44.259103 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:44.259198 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 09:31:44.259219 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:31:44.259433 22976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:44.255904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58890 (local address 127.0.98.61:42775)
0926 09:31:44.255945 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:44.255947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:44.256213 (+ 266us) server_negotiation.cc:407] Connection header received
0926 09:31:44.256245 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:44.256246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:44.256258 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:44.256330 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:44.256481 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:44.257256 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:44.258548 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:44.258849 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:44.258919 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:44.259127 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:44.259161 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:44.259194 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:44.259333 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 09:31:44.259366 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:31:44.260782 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 22574
I20260926 09:31:44.265846 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 22649
I20260926 09:31:44.270771 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 22724
2026-09-26T09:31:44Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2867 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:31:44 dist-test-slave-dfzm krb5kdc[22993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:31:44 dist-test-slave-dfzm krb5kdc[22993](info): set up 2 sockets
Sep 26 09:31:44 dist-test-slave-dfzm krb5kdc[22993](info): commencing operation
krb5kdc: starting...
W20260926 09:31:45.309966 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/krb5kdc/test-user.keytab.
Sep 26 09:31:45 dist-test-slave-dfzm krb5kdc[22993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:31:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:31:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 09:31:45.429790 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38155
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38155,127.0.98.61:36207,127.0.98.60:42287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:31:45.510787 23006 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 09:31:45.510999 23006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:45.511018 23006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:45.511044 23006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:45.512125 23006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:45.512163 23006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:45.512177 23006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:45.512189 23006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:45.512202 23006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:45.513633 23006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38155,127.0.98.61:36207,127.0.98.60:42287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:45.513830 23006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:45.513998 23006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:45.518779 23011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:45.518775 23012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:45.518805 23014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:45.518929 23006 server_base.cc:1034] running on GCE node
I20260926 09:31:45.519271 23006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:45.519627 23006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:45.520788 23006 hybrid_clock.cc:648] HybridClock initialized: now 1790415105520770 us; error 38 us; skew 500 ppm
Sep 26 09:31:45 dist-test-slave-dfzm krb5kdc[22993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:45.523362 23006 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 09:31:45.523782 23006 webserver.cc:460] Webserver started at http://127.0.98.62:42377/ using document root <none> and password file <none>
I20260926 09:31:45.523967 23006 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:45.524008 23006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:45.524106 23006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:45.525199 23006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-0/data/instance:
uuid: "6b75bc97a2084e62a6c1d41b92ada830"
format_stamp: "Formatted at 2026-09-26 09:31:45 on dist-test-slave-dfzm"
server_key: "db85517665d74e6859212aa1784debea"
server_key_iv: "93cbbc5202734128030c80c99b2227b4"
server_key_version: "encryptionkey@0"
I20260926 09:31:45.525498 23006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-0/wal/instance:
uuid: "6b75bc97a2084e62a6c1d41b92ada830"
format_stamp: "Formatted at 2026-09-26 09:31:45 on dist-test-slave-dfzm"
server_key: "db85517665d74e6859212aa1784debea"
server_key_iv: "93cbbc5202734128030c80c99b2227b4"
server_key_version: "encryptionkey@0"
I20260926 09:31:45.526894 23006 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:31:45.527659 23021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:45.527833 23006 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:31:45.527922 23006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-0/wal
uuid: "6b75bc97a2084e62a6c1d41b92ada830"
format_stamp: "Formatted at 2026-09-26 09:31:45 on dist-test-slave-dfzm"
server_key: "db85517665d74e6859212aa1784debea"
server_key_iv: "93cbbc5202734128030c80c99b2227b4"
server_key_version: "encryptionkey@0"
I20260926 09:31:45.527997 23006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:45.537108 23006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:45.539001 23006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:45.539161 23006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:45.542976 23006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38155
I20260926 09:31:45.543025 23073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38155 every 8 connection(s)
I20260926 09:31:45.543401 23006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-0/data/info.pb
I20260926 09:31:45.544423 23074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:45.544667 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 23006
I20260926 09:31:45.544770 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-0/wal/instance
I20260926 09:31:45.544957 392 external_mini_cluster.cc:1428] Setting key f1af7b5c4ffd6442730b008b5267c1c0
I20260926 09:31:45.546131 23074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Sep 26 09:31:45 dist-test-slave-dfzm krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260926 09:31:45.555630 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.546763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38155 (local address 127.0.0.1:55642)
0926 09:31:45.546928 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.546972 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:45.547427 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:45.547742 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:45.547744 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:45.548030 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:45.548181 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:45.548185 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.549303 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.549304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:45.550069 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:45.550071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.550561 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.550563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:45.550625 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.551150 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:45.551158 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:45.552495 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:45.554136 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:45.554142 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:45.554144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:45.554364 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:45.554581 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:45.554582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:45.554583 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:45.554669 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:45.555006 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:45.555011 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:45.555071 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 09:31:45.555174 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 09:31:45.555392 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":83,"thread_start_us":81,"threads_started":1}
I20260926 09:31:45.555634 23078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.546952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55642 (local address 127.0.98.62:38155)
0926 09:31:45.547092 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:45.547098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:45.547136 (+ 38us) server_negotiation.cc:407] Connection header received
0926 09:31:45.547588 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:45.547590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:45.547619 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:45.547726 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:45.548288 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.549206 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.550152 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.550454 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.550577 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.552576 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:45.552583 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:45.552584 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:45.552590 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:45.554046 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.554438 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.554441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.554442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.554517 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.554730 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.554731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.554731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.554940 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:45.555028 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:45.555197 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 09:31:45.555331 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260926 09:31:45.556372 23074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:31:45.556797 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.556652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36207 (local address 127.0.0.1:41842)
0926 09:31:45.556724 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.556772 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 09:31:45.556977 23025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36207: connect: Connection refused (error 111)
I20260926 09:31:45.557402 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.557317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36207 (local address 127.0.0.1:41846)
0926 09:31:45.557358 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.557384 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 09:31:45.557528 23074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36207: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36207: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 09:31:45.568756 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36207
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38155,127.0.98.61:36207,127.0.98.60:42287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:31:45.606788 23074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 } attempt: 1
I20260926 09:31:45.607308 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.607167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36207 (local address 127.0.0.1:41862)
0926 09:31:45.607232 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.607271 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 09:31:45.607664 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.607562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36207 (local address 127.0.0.1:41874)
0926 09:31:45.607614 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.607645 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 09:31:45.607750 23074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36207: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36207: connect: Connection refused (error 111)
W20260926 09:31:45.650151 23081 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 09:31:45.650418 23081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:45.650460 23081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:45.650519 23081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:45.651801 23081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:45.651875 23081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:45.651901 23081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:45.651932 23081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:45.651958 23081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:45.653657 23081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38155,127.0.98.61:36207,127.0.98.60:42287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:45.653942 23081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:45.654173 23081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:45.659027 23089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:45.659081 23086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:45.659261 23087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:45.659232 23081 server_base.cc:1034] running on GCE node
I20260926 09:31:45.659502 23081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:45.659751 23081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:45.660929 23081 hybrid_clock.cc:648] HybridClock initialized: now 1790415105660920 us; error 34 us; skew 500 ppm
Sep 26 09:31:45 dist-test-slave-dfzm krb5kdc[22993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:45.663409 23081 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 09:31:45.663906 23081 webserver.cc:460] Webserver started at http://127.0.98.61:34723/ using document root <none> and password file <none>
I20260926 09:31:45.664135 23081 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:45.664201 23081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:45.664348 23081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:45.665405 23081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-1/data/instance:
uuid: "8eacc667f7ce447f964655e8a7e8ce40"
format_stamp: "Formatted at 2026-09-26 09:31:45 on dist-test-slave-dfzm"
server_key: "66dc8d8e0fd5ad354e4a8b15b3f0264d"
server_key_iv: "5d0e7527028d6faa7685be610bec8e55"
server_key_version: "encryptionkey@0"
I20260926 09:31:45.665791 23081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-1/wal/instance:
uuid: "8eacc667f7ce447f964655e8a7e8ce40"
format_stamp: "Formatted at 2026-09-26 09:31:45 on dist-test-slave-dfzm"
server_key: "66dc8d8e0fd5ad354e4a8b15b3f0264d"
server_key_iv: "5d0e7527028d6faa7685be610bec8e55"
server_key_version: "encryptionkey@0"
I20260926 09:31:45.667081 23081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:31:45.667752 23096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:45.667930 23081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:31:45.668027 23081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-1/wal
uuid: "8eacc667f7ce447f964655e8a7e8ce40"
format_stamp: "Formatted at 2026-09-26 09:31:45 on dist-test-slave-dfzm"
server_key: "66dc8d8e0fd5ad354e4a8b15b3f0264d"
server_key_iv: "5d0e7527028d6faa7685be610bec8e55"
server_key_version: "encryptionkey@0"
I20260926 09:31:45.668119 23081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:45.675881 23074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 } attempt: 2
I20260926 09:31:45.676313 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.676187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36207 (local address 127.0.0.1:41888)
0926 09:31:45.676245 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.676283 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 09:31:45.676646 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.676550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36207 (local address 127.0.0.1:41902)
0926 09:31:45.676594 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.676620 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 09:31:45.676739 23074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36207: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36207: connect: Connection refused (error 111)
I20260926 09:31:45.687448 23081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:45.689244 23081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:45.689404 23081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:45.693437 23081 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36207
I20260926 09:31:45.693499 23148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36207 every 8 connection(s)
I20260926 09:31:45.693897 23081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-1/data/info.pb
I20260926 09:31:45.694806 23149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:45.696283 23149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:31:45 dist-test-slave-dfzm krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:31:45.703738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 23081
I20260926 09:31:45.703852 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-1/wal/instance
I20260926 09:31:45.704087 392 external_mini_cluster.cc:1428] Setting key 4cf6a7a425ff871f6460a13f99da0c67
I20260926 09:31:45.704952 23078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.696918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55656 (local address 127.0.98.62:38155)
0926 09:31:45.696980 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:45.696982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:45.697171 (+ 189us) server_negotiation.cc:407] Connection header received
0926 09:31:45.697702 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:45.697703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:45.697719 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:45.697773 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:45.698405 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.698861 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.699795 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.700001 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.700054 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.702147 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:45.702149 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:45.702149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:45.702154 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:45.703344 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.703838 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.703838 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.703839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.703945 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.704224 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.704225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.704225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.704340 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:45.704400 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:45.704793 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 09:31:45.704889 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:31:45.705303 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.696895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38155 (local address 127.0.0.1:55656)
0926 09:31:45.697070 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.697105 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:45.697545 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:45.697844 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:45.697847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:45.698136 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:45.698328 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:45.698332 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.698943 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.698943 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:45.699726 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:45.699728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.700082 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.700083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:45.700185 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.700697 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:45.700705 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:45.702070 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:45.703414 (+ 1344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:45.703421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:45.703422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:45.703750 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:45.704001 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:45.704002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:45.704003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:45.704152 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:45.704407 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:45.704413 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:45.704581 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 09:31:45.704709 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 09:31:45.704887 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
I20260926 09:31:45.706075 23149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:31:45 dist-test-slave-dfzm krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260926 09:31:45.714493 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.706357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36207 (local address 127.0.0.1:41912)
0926 09:31:45.706416 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.706426 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:45.706473 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:45.706714 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:45.706715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:45.706855 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:45.706898 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:45.706900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.708209 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.708210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:45.708794 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:45.708797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.709311 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.709313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:45.709419 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.709929 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:45.709935 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:45.711325 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:45.713341 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:45.713342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:45.713342 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:45.713509 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:45.713743 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:45.713744 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:45.713745 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:45.713799 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:45.714110 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:45.714110 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:45.714149 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 09:31:45.714225 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 09:31:45.714335 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 09:31:45.714556 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.706408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41912 (local address 127.0.98.61:36207)
0926 09:31:45.706579 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:45.706582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:45.706598 (+ 16us) server_negotiation.cc:407] Connection header received
0926 09:31:45.706613 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:45.706615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:45.706625 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:45.706711 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:45.706968 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.708121 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.708875 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.709232 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.709290 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.711439 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:45.711448 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:45.711448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:45.711455 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:45.713242 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.713619 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.713623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.713624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.713678 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.713865 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.713866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.713867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.714021 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:45.714130 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:45.714260 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 09:31:45.714387 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":142,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260926 09:31:45.715374 23149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:31:45.715765 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.715625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42287 (local address 127.0.0.1:34956)
0926 09:31:45.715672 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.715732 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 09:31:45.715929 23100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42287: connect: Connection refused (error 111)
I20260926 09:31:45.716357 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.716251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42287 (local address 127.0.0.1:34960)
0926 09:31:45.716295 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.716326 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 09:31:45.717145 23149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42287: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42287: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 09:31:45.732254 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
/tmp/dist-test-taskNX8gJ6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42287
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38155,127.0.98.61:36207,127.0.98.60:42287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:31:45.766381 23149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } attempt: 1
I20260926 09:31:45.766950 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.766732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42287 (local address 127.0.0.1:34966)
0926 09:31:45.766803 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.766857 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 09:31:45.767336 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.767239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42287 (local address 127.0.0.1:34970)
0926 09:31:45.767285 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.767316 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 09:31:45.767468 23149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42287: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42287: connect: Connection refused (error 111)
I20260926 09:31:45.767865 23074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 } attempt: 3
Sep 26 09:31:45 dist-test-slave-dfzm krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 09:31:45.774966 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.768195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41914 (local address 127.0.98.61:36207)
0926 09:31:45.768256 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:45.768259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:45.768269 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:31:45.768308 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:45.768308 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:45.768323 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:45.768378 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:45.768697 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.769214 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.770006 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.770220 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.770293 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.772456 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:45.772457 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:45.772457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:45.772464 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:45.773920 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.774296 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.774297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.774297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.774348 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.774538 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.774538 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.774539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.774641 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:45.774689 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:45.774841 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 09:31:45.774903 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 09:31:45.775019 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.768117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36207 (local address 127.0.0.1:41914)
0926 09:31:45.768169 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.768180 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:45.768243 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:45.768385 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:45.768386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:45.768553 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:45.768617 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:45.768619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.769316 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.769317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:45.769906 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:45.769908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.770301 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.770302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:45.770380 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.770885 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:45.770891 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:45.772374 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:45.773997 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:45.773998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:45.773998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:45.774225 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:45.774410 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:45.774411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:45.774411 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:45.774480 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:45.774696 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:45.774697 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:45.774749 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 09:31:45.774818 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 09:31:45.774925 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 09:31:45.775463 23074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:31:45.775854 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.775727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42287 (local address 127.0.0.1:34984)
0926 09:31:45.775760 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.775827 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 09:31:45.776126 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.776069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42287 (local address 127.0.0.1:34986)
0926 09:31:45.776092 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.776113 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260926 09:31:45.776275 23074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42287: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42287: connect: Connection refused (error 111)
I20260926 09:31:45.807418 23074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } attempt: 1
I20260926 09:31:45.807994 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.807808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42287 (local address 127.0.0.1:34990)
0926 09:31:45.807905 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.807962 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 09:31:45.808399 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.808314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42287 (local address 127.0.0.1:35002)
0926 09:31:45.808353 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.808380 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 09:31:45.808502 23074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42287: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42287: connect: Connection refused (error 111)
W20260926 09:31:45.815920 23156 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 09:31:45.816133 23156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:31:45.816150 23156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:31:45.816213 23156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:31:45.817356 23156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:31:45.817395 23156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:31:45.817409 23156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:31:45.817422 23156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:31:45.817457 23156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:31:45.818977 23156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38155,127.0.98.61:36207,127.0.98.60:42287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:31:45.819231 23156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:31:45.819463 23156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:31:45.824146 23164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:45.824201 23162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:31:45.824165 23161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:31:45.824366 23156 server_base.cc:1034] running on GCE node
I20260926 09:31:45.824606 23156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:31:45.824887 23156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:31:45.826059 23156 hybrid_clock.cc:648] HybridClock initialized: now 1790415105826050 us; error 34 us; skew 500 ppm
Sep 26 09:31:45 dist-test-slave-dfzm krb5kdc[22993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:31:45.828637 23156 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 09:31:45.829087 23156 webserver.cc:460] Webserver started at http://127.0.98.60:33521/ using document root <none> and password file <none>
I20260926 09:31:45.829313 23156 fs_manager.cc:359] Metadata directory not provided
I20260926 09:31:45.829380 23156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:31:45.829522 23156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:31:45.830623 23156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-2/data/instance:
uuid: "ec5da7d4a27a4cc1a986b8d3bd4138aa"
format_stamp: "Formatted at 2026-09-26 09:31:45 on dist-test-slave-dfzm"
server_key: "f2ef34f0ecd70e278e776dfcf1cd7904"
server_key_iv: "f3541a2aa3ba60a631750c6aac486a3d"
server_key_version: "encryptionkey@0"
I20260926 09:31:45.830979 23156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-2/wal/instance:
uuid: "ec5da7d4a27a4cc1a986b8d3bd4138aa"
format_stamp: "Formatted at 2026-09-26 09:31:45 on dist-test-slave-dfzm"
server_key: "f2ef34f0ecd70e278e776dfcf1cd7904"
server_key_iv: "f3541a2aa3ba60a631750c6aac486a3d"
server_key_version: "encryptionkey@0"
I20260926 09:31:45.832365 23156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:31:45.833158 23171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:31:45.833338 23156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:31:45.833437 23156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-2/wal
uuid: "ec5da7d4a27a4cc1a986b8d3bd4138aa"
format_stamp: "Formatted at 2026-09-26 09:31:45 on dist-test-slave-dfzm"
server_key: "f2ef34f0ecd70e278e776dfcf1cd7904"
server_key_iv: "f3541a2aa3ba60a631750c6aac486a3d"
server_key_version: "encryptionkey@0"
I20260926 09:31:45.833530 23156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:31:45.835656 23149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } attempt: 2
I20260926 09:31:45.836072 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.835949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42287 (local address 127.0.0.1:35014)
0926 09:31:45.836004 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.836041 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 09:31:45.836494 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.836406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42287 (local address 127.0.0.1:35028)
0926 09:31:45.836449 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.836475 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 09:31:45.836653 23149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42287: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42287: connect: Connection refused (error 111)
I20260926 09:31:45.846040 23156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:31:45.848865 23156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:31:45.849041 23156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:31:45.852969 23156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42287
I20260926 09:31:45.853039 23223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42287 every 8 connection(s)
I20260926 09:31:45.853374 23156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-2/data/info.pb
I20260926 09:31:45.854414 23224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:31:45.855944 23224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:31:45.856968 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu as pid 23156
I20260926 09:31:45.857081 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNX8gJ6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414839069479-392-0/minicluster-data/master-2/wal/instance
I20260926 09:31:45.857263 392 external_mini_cluster.cc:1428] Setting key d8c51edac6fd240da45d47d6dbe7532e
Sep 26 09:31:45 dist-test-slave-dfzm krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 09:31:45 dist-test-slave-dfzm krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 09:31:45.866027 23078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.856684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55662 (local address 127.0.98.62:38155)
0926 09:31:45.856742 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:45.856744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:45.856959 (+ 215us) server_negotiation.cc:407] Connection header received
0926 09:31:45.857702 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:45.857704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:45.857718 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:45.857789 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:45.858410 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.858881 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.859944 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.860153 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.860238 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.862688 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:45.862690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:45.862690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:45.862697 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:45.864444 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.864920 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.864922 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.864922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.864997 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.865265 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.865266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.865267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.865397 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:45.865480 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:45.865833 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 09:31:45.865922 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 09:31:45.866276 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.856658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38155 (local address 127.0.0.1:55662)
0926 09:31:45.856823 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.856866 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:45.857514 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:45.857919 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:45.857923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:45.858167 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:45.858317 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:45.858321 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.858969 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.858970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:45.859841 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:45.859844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.860262 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.860263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:45.860377 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.860909 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:45.860919 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:45.862601 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:45.864574 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:45.864580 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:45.864582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:45.864832 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:45.865077 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:45.865078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:45.865078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:45.865173 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:45.865466 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:45.865473 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:45.865664 (+ 191us) client_negotiation.cc:769] Sending connection context
0926 09:31:45.865796 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 09:31:45.865941 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":144,"thread_start_us":91,"threads_started":1}
I20260926 09:31:45.866564 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.857693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35038 (local address 127.0.98.60:42287)
0926 09:31:45.857860 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:45.857865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:45.857905 (+ 40us) server_negotiation.cc:407] Connection header received
0926 09:31:45.857928 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:45.857930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:45.857956 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:45.858045 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:45.858665 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.859465 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.860202 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.860496 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.860572 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.863248 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:45.863256 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:45.863256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:45.863263 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:45.865152 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.865501 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.865504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.865504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.865570 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.865817 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.865818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.865818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.865999 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:45.866172 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:45.866226 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 09:31:45.866373 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260926 09:31:45.866838 23224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:31:45 dist-test-slave-dfzm krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 09:31:45 dist-test-slave-dfzm krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 09:31:45.873984 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.867101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36207 (local address 127.0.0.1:41922)
0926 09:31:45.867136 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.867147 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:45.867190 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:45.867395 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:45.867397 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:45.867528 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:45.867561 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:45.867562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.868265 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.868266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:45.868794 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:45.868795 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.869208 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.869216 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:45.869279 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.869694 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:45.869699 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:45.871042 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:45.872978 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:45.872980 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:45.872980 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:45.873191 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:45.873379 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:45.873380 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:45.873381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:45.873447 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:45.873739 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:45.873740 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:45.873787 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 09:31:45.873858 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 09:31:45.873922 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 09:31:45.874048 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.867172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41922 (local address 127.0.98.61:36207)
0926 09:31:45.867228 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:45.867231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:45.867243 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:31:45.867270 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:45.867271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:45.867284 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:45.867346 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:45.867645 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.868158 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.868900 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.869123 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.869210 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.871123 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:45.871125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:45.871125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:45.871134 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:45.872896 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.873262 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.873263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.873263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.873316 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.873511 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.873512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.873512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.873670 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:45.873737 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:45.873888 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 09:31:45.873948 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:31:45.874380 23224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:31:45.875218 23078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.867302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55676 (local address 127.0.98.62:38155)
0926 09:31:45.867349 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:45.867351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:45.867361 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:31:45.867430 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:45.867431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:45.867441 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:45.867495 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:45.867925 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.868414 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.869080 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.869278 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.869348 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.872028 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:45.872029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:45.872030 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:45.872037 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:45.873673 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.874367 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.874368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.874368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.874421 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.874711 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.874712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.874712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.874855 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:45.875043 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:45.875086 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 09:31:45.875160 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Sep 26 09:31:45 dist-test-slave-dfzm krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 09:31:45 dist-test-slave-dfzm krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 09:31:45.881464 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.874636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42287 (local address 127.0.0.1:35054)
0926 09:31:45.874676 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.874686 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:45.874720 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:45.874839 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:45.874840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:45.874979 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:45.875014 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:45.875016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.876116 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.876117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:45.876583 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:45.876584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.877040 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.877041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:45.877078 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.877483 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:45.877487 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:45.878690 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:45.880360 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:45.880361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:45.880362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:45.880578 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:45.880843 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:45.880844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:45.880844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:45.880901 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:45.881176 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:45.881177 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:45.881228 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 09:31:45.881301 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 09:31:45.881397 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 09:31:45.881608 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.874670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35054 (local address 127.0.98.60:42287)
0926 09:31:45.874703 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:45.874705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:45.874712 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:31:45.874775 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:45.874776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:45.874784 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:45.874819 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:45.875086 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.875783 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.876662 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.876968 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.877037 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.878781 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:45.878782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:45.878783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:45.878789 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:45.880305 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.880658 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.880659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.880659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.880704 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.880965 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.880967 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.880968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.881112 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:45.881165 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:45.881351 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 09:31:45.881461 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":16}
I20260926 09:31:45.882906 23224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec5da7d4a27a4cc1a986b8d3bd4138aa: Bootstrap starting.
I20260926 09:31:45.883255 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.875796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41924 (local address 127.0.98.61:36207)
0926 09:31:45.875851 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:45.875854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:45.875863 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:31:45.875883 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:45.875884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:45.875893 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:45.875955 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:45.876409 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.877026 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.877895 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.878095 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.878165 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.880307 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:45.880308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:45.880308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:45.880314 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:45.882173 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.882529 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.882530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.882530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.882582 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.882779 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.882780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.882781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.882890 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:45.883048 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:45.883084 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 09:31:45.883151 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 09:31:45.883625 23074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } attempt: 2
I20260926 09:31:45.884006 23224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec5da7d4a27a4cc1a986b8d3bd4138aa: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:45.884334 23224 log.cc:824] T 00000000000000000000000000000000 P ec5da7d4a27a4cc1a986b8d3bd4138aa: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:45.884860 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:31:45.885901 23224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec5da7d4a27a4cc1a986b8d3bd4138aa: No bootstrap required, opened a new log
I20260926 09:31:45.886006 23048 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6b75bc97a2084e62a6c1d41b92ada830"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55676
I20260926 09:31:45.887718 23224 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec5da7d4a27a4cc1a986b8d3bd4138aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b75bc97a2084e62a6c1d41b92ada830" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } } peers { permanent_uuid: "8eacc667f7ce447f964655e8a7e8ce40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 } } peers { permanent_uuid: "ec5da7d4a27a4cc1a986b8d3bd4138aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } }
I20260926 09:31:45.887861 23224 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec5da7d4a27a4cc1a986b8d3bd4138aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec5da7d4a27a4cc1a986b8d3bd4138aa, State: Initialized, Role: FOLLOWER
Sep 26 09:31:45 dist-test-slave-dfzm krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 09:31:45.887965 23224 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec5da7d4a27a4cc1a986b8d3bd4138aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b75bc97a2084e62a6c1d41b92ada830" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } } peers { permanent_uuid: "8eacc667f7ce447f964655e8a7e8ce40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 } } peers { permanent_uuid: "ec5da7d4a27a4cc1a986b8d3bd4138aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } }
I20260926 09:31:45.888237 23230 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec5da7d4a27a4cc1a986b8d3bd4138aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b75bc97a2084e62a6c1d41b92ada830" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } } peers { permanent_uuid: "8eacc667f7ce447f964655e8a7e8ce40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 } } peers { permanent_uuid: "ec5da7d4a27a4cc1a986b8d3bd4138aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } } }
I20260926 09:31:45.888298 23224 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec5da7d4a27a4cc1a986b8d3bd4138aa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:45.888386 23230 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec5da7d4a27a4cc1a986b8d3bd4138aa [sys.catalog]: This master's current role is: FOLLOWER
W20260926 09:31:45.890215 23241 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec5da7d4a27a4cc1a986b8d3bd4138aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:31:45.890322 23241 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:31:45.891265 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.883927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35058 (local address 127.0.98.60:42287)
0926 09:31:45.884015 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:45.884017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:45.884025 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:31:45.884042 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:45.884043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:45.884053 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:45.884438 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:45.884478 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.885131 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.886071 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.886330 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.886407 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.888363 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:45.888364 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:45.888364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:45.888371 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:45.889932 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.890490 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.890491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.890492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.890552 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.890737 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.890737 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.890738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.890887 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:45.890944 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:45.891114 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 09:31:45.891180 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 09:31:45.891268 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.883904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42287 (local address 127.0.0.1:35058)
0926 09:31:45.883942 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.883953 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:45.883991 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:45.884149 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:45.884151 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:45.884319 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:45.884366 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:45.884368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.885231 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.885232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:45.885995 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:45.885998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.886412 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.886413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:45.886470 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.886887 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:45.886893 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:45.888260 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:45.890025 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:45.890026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:45.890027 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:45.890248 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:45.890613 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:45.890614 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:45.890614 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:45.890688 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:45.890956 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:45.890957 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:45.890999 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 09:31:45.891075 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 09:31:45.891174 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 09:31:45.892513 23074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830: Bootstrap starting.
I20260926 09:31:45.893028 23074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:45.893234 23074 log.cc:824] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:45.890604 23156 master_runner.cc:186] Error getting master registration for 127.0.98.62:38155: OK, instance_id { permanent_uuid: "6b75bc97a2084e62a6c1d41b92ada830" instance_seqno: 1790415105541911 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 09:31:45.893399 23156 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 09:31:45.894009 23074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830: No bootstrap required, opened a new log
I20260926 09:31:45.894948 23074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b75bc97a2084e62a6c1d41b92ada830" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } } peers { permanent_uuid: "8eacc667f7ce447f964655e8a7e8ce40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 } } peers { permanent_uuid: "ec5da7d4a27a4cc1a986b8d3bd4138aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } }
I20260926 09:31:45.895056 23074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b75bc97a2084e62a6c1d41b92ada830, State: Initialized, Role: FOLLOWER
I20260926 09:31:45.895133 23074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b75bc97a2084e62a6c1d41b92ada830" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } } peers { permanent_uuid: "8eacc667f7ce447f964655e8a7e8ce40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 } } peers { permanent_uuid: "ec5da7d4a27a4cc1a986b8d3bd4138aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } }
I20260926 09:31:45.895376 23244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b75bc97a2084e62a6c1d41b92ada830" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } } peers { permanent_uuid: "8eacc667f7ce447f964655e8a7e8ce40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 } } peers { permanent_uuid: "ec5da7d4a27a4cc1a986b8d3bd4138aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } } }
I20260926 09:31:45.895442 23074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:45.895452 23244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:31:45.896725 23048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 09:31:45.896986 23048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830 [term 0 FOLLOWER]: Advancing to term 1
W20260926 09:31:45.897159 23255 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:31:45.897200 23255 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:31:45.897842 23048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b75bc97a2084e62a6c1d41b92ada830" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } } peers { permanent_uuid: "8eacc667f7ce447f964655e8a7e8ce40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 } } peers { permanent_uuid: "ec5da7d4a27a4cc1a986b8d3bd4138aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } }
I20260926 09:31:45.898079 23048 leader_election.cc:288] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830 [CANDIDATE]: Term 1 election: Requested vote from peers 8eacc667f7ce447f964655e8a7e8ce40 (127.0.98.61:36207), ec5da7d4a27a4cc1a986b8d3bd4138aa (127.0.98.60:42287)
I20260926 09:31:45.898392 23198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b75bc97a2084e62a6c1d41b92ada830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ec5da7d4a27a4cc1a986b8d3bd4138aa"
I20260926 09:31:45.898511 23198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec5da7d4a27a4cc1a986b8d3bd4138aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:45.899835 23198 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ec5da7d4a27a4cc1a986b8d3bd4138aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b75bc97a2084e62a6c1d41b92ada830 in term 1.
I20260926 09:31:45.900068 23025 leader_election.cc:302] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b75bc97a2084e62a6c1d41b92ada830, ec5da7d4a27a4cc1a986b8d3bd4138aa; no voters:
I20260926 09:31:45.900238 23244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:31:45.900384 23244 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830 [term 1 LEADER]: Becoming Leader. State: Replica: 6b75bc97a2084e62a6c1d41b92ada830, State: Running, Role: LEADER
I20260926 09:31:45.900460 23244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b75bc97a2084e62a6c1d41b92ada830" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } } peers { permanent_uuid: "8eacc667f7ce447f964655e8a7e8ce40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 } } peers { permanent_uuid: "ec5da7d4a27a4cc1a986b8d3bd4138aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } }
I20260926 09:31:45.901248 23006 master_runner.cc:186] Error getting master registration for 127.0.98.61:36207: OK, instance_id { permanent_uuid: "8eacc667f7ce447f964655e8a7e8ce40" instance_seqno: 1790415105692350 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 09:31:45.903543 23006 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 09:31:45.903833 23261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b75bc97a2084e62a6c1d41b92ada830. Latest consensus state: current_term: 1 leader_uuid: "6b75bc97a2084e62a6c1d41b92ada830" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b75bc97a2084e62a6c1d41b92ada830" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } } peers { permanent_uuid: "8eacc667f7ce447f964655e8a7e8ce40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 } } peers { permanent_uuid: "ec5da7d4a27a4cc1a986b8d3bd4138aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } } }
I20260926 09:31:45.903929 23261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:45.906576 23263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:31:45.906832 23263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:31:45.911007 23198 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec5da7d4a27a4cc1a986b8d3bd4138aa [term 1 FOLLOWER]: Refusing update from remote peer 6b75bc97a2084e62a6c1d41b92ada830: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:31:45.911398 23261 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec5da7d4a27a4cc1a986b8d3bd4138aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:31:45.912722 23230 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec5da7d4a27a4cc1a986b8d3bd4138aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b75bc97a2084e62a6c1d41b92ada830. Latest consensus state: current_term: 1 leader_uuid: "6b75bc97a2084e62a6c1d41b92ada830" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b75bc97a2084e62a6c1d41b92ada830" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } } peers { permanent_uuid: "8eacc667f7ce447f964655e8a7e8ce40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 } } peers { permanent_uuid: "ec5da7d4a27a4cc1a986b8d3bd4138aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } } }
I20260926 09:31:45.912825 23230 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec5da7d4a27a4cc1a986b8d3bd4138aa [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:31:45.913771 23261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b75bc97a2084e62a6c1d41b92ada830" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b75bc97a2084e62a6c1d41b92ada830" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } } peers { permanent_uuid: "8eacc667f7ce447f964655e8a7e8ce40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 } } peers { permanent_uuid: "ec5da7d4a27a4cc1a986b8d3bd4138aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } } }
I20260926 09:31:45.913851 23261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:45.914170 23230 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec5da7d4a27a4cc1a986b8d3bd4138aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b75bc97a2084e62a6c1d41b92ada830" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b75bc97a2084e62a6c1d41b92ada830" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } } peers { permanent_uuid: "8eacc667f7ce447f964655e8a7e8ce40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 } } peers { permanent_uuid: "ec5da7d4a27a4cc1a986b8d3bd4138aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } } }
I20260926 09:31:45.914251 23230 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec5da7d4a27a4cc1a986b8d3bd4138aa [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:31:45.914615 23267 mvcc.cc:205] Tried to move back new op lower bound from 7333540273809047552 to 7333540273769504768. Current Snapshot: MvccSnapshot[applied={T|T < 7333540273809047552}]
I20260926 09:31:45.915076 23268 mvcc.cc:205] Tried to move back new op lower bound from 7333540273809047552 to 7333540273769504768. Current Snapshot: MvccSnapshot[applied={T|T < 7333540273809047552}]
I20260926 09:31:45.915625 23263 catalog_manager.cc:1347] Generated new cluster ID: 4b3c9ee8a76e4ad3b5e693a4cc1f7e9d
I20260926 09:31:45.915676 23263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:31:45.919134 23078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.906162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55692 (local address 127.0.98.62:38155)
0926 09:31:45.906215 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:45.906217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:45.909966 (+ 3749us) server_negotiation.cc:407] Connection header received
0926 09:31:45.909991 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:45.909993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:45.910004 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:45.910052 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:45.911487 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.912144 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.913641 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.913954 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.913998 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.916139 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:45.916141 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:45.916141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:45.916148 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:45.917680 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.918192 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.918193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.918194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.918255 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.918492 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.918492 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.918493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.918607 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:45.918685 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:45.918973 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 09:31:45.919045 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:31:45.921039 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.905926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41938 (local address 127.0.98.61:36207)
0926 09:31:45.905987 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:45.905989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:45.906631 (+ 642us) server_negotiation.cc:407] Connection header received
0926 09:31:45.906853 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:45.906854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:45.906866 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:45.906897 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:45.908183 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.908932 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.910565 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.910889 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.910933 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.913701 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:45.913703 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:45.913703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:45.913712 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:45.916183 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.920071 (+ 3888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.920072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.920072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.920144 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.920420 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.920421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.920421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.920560 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:45.920642 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:45.920876 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 09:31:45.920953 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:31:45.927784 23149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } attempt: 3
I20260926 09:31:45.928560 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.906365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35066 (local address 127.0.98.60:42287)
0926 09:31:45.906408 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:45.906410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:45.909771 (+ 3361us) server_negotiation.cc:407] Connection header received
0926 09:31:45.909856 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:45.909857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:45.909869 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:45.909927 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:45.919522 (+ 9595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.920226 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.923144 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.923416 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.923484 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.925365 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:45.925366 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:45.925376 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:45.925383 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:45.926892 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.927426 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.927427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.927427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.927476 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.927708 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.927709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.927709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.927838 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:45.927935 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:45.928392 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 09:31:45.928469 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 26 09:31:45 dist-test-slave-dfzm krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790415105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 09:31:45.935231 23263 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:31:45.936298 23263 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:31:45.939781 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.928061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42287 (local address 127.0.0.1:35080)
0926 09:31:45.928164 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 09:31:45.928175 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:31:45.928214 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:31:45.928845 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:31:45.928846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:31:45.929001 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:31:45.929046 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:45.929048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.929940 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.929941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:45.930740 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:31:45.930742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.931371 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.931372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:31:45.931436 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.931919 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:31:45.931924 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:31:45.933334 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:31:45.938582 (+ 5248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:45.938583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:45.938584 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:45.938794 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:45.939072 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:31:45.939073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:31:45.939073 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:31:45.939135 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:31:45.939490 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:31:45.939491 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:31:45.939544 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 09:31:45.939588 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 09:31:45.939686 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 09:31:45.940336 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.928175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35080 (local address 127.0.98.60:42287)
0926 09:31:45.928682 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:45.928684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:45.928690 (+ 6us) server_negotiation.cc:407] Connection header received
0926 09:31:45.928718 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:45.928719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:45.928728 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:45.929114 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:45.929151 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.929848 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.930863 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.931276 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.931611 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.933434 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:31:45.933436 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:31:45.933436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:31:45.933444 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:31:45.938477 (+ 5033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.938881 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.938882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.938882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.938989 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:31:45.939205 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:31:45.939205 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:31:45.939205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:31:45.939406 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:31:45.940000 (+ 594us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:45.940064 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 09:31:45.940183 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":98,"threads_started":1}
I20260926 09:31:45.941804 23149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eacc667f7ce447f964655e8a7e8ce40: Bootstrap starting.
I20260926 09:31:45.942487 23149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8eacc667f7ce447f964655e8a7e8ce40: Neither blocks nor log segments found. Creating new log.
I20260926 09:31:45.942762 23149 log.cc:824] T 00000000000000000000000000000000 P 8eacc667f7ce447f964655e8a7e8ce40: Log is configured to *not* fsync() on all Append() calls
I20260926 09:31:45.954403 23149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eacc667f7ce447f964655e8a7e8ce40: No bootstrap required, opened a new log
I20260926 09:31:45.956566 23263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830: Generated new TSK 0
I20260926 09:31:45.956995 23263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:31:45.958233 23149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8eacc667f7ce447f964655e8a7e8ce40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b75bc97a2084e62a6c1d41b92ada830" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } } peers { permanent_uuid: "8eacc667f7ce447f964655e8a7e8ce40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 } } peers { permanent_uuid: "ec5da7d4a27a4cc1a986b8d3bd4138aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } }
I20260926 09:31:45.958400 23149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8eacc667f7ce447f964655e8a7e8ce40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eacc667f7ce447f964655e8a7e8ce40, State: Initialized, Role: FOLLOWER
I20260926 09:31:45.958493 23149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8eacc667f7ce447f964655e8a7e8ce40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b75bc97a2084e62a6c1d41b92ada830" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } } peers { permanent_uuid: "8eacc667f7ce447f964655e8a7e8ce40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 } } peers { permanent_uuid: "ec5da7d4a27a4cc1a986b8d3bd4138aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } }
I20260926 09:31:45.958855 23149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8eacc667f7ce447f964655e8a7e8ce40 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:31:45.958926 23122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8eacc667f7ce447f964655e8a7e8ce40 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:31:45.959969 23122 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8eacc667f7ce447f964655e8a7e8ce40 [term 1 FOLLOWER]: Refusing update from remote peer 6b75bc97a2084e62a6c1d41b92ada830: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:31:45.958830 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eacc667f7ce447f964655e8a7e8ce40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b75bc97a2084e62a6c1d41b92ada830" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } } peers { permanent_uuid: "8eacc667f7ce447f964655e8a7e8ce40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 } } peers { permanent_uuid: "ec5da7d4a27a4cc1a986b8d3bd4138aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } } }
I20260926 09:31:45.960410 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eacc667f7ce447f964655e8a7e8ce40 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:31:45.960184 23123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b75bc97a2084e62a6c1d41b92ada830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8eacc667f7ce447f964655e8a7e8ce40"
I20260926 09:31:45.961318 23123 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8eacc667f7ce447f964655e8a7e8ce40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b75bc97a2084e62a6c1d41b92ada830 in term 1.
I20260926 09:31:45.961663 23244 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eacc667f7ce447f964655e8a7e8ce40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:31:45.965521 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eacc667f7ce447f964655e8a7e8ce40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b75bc97a2084e62a6c1d41b92ada830. Latest consensus state: current_term: 1 leader_uuid: "6b75bc97a2084e62a6c1d41b92ada830" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b75bc97a2084e62a6c1d41b92ada830" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } } peers { permanent_uuid: "8eacc667f7ce447f964655e8a7e8ce40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 } } peers { permanent_uuid: "ec5da7d4a27a4cc1a986b8d3bd4138aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } } }
I20260926 09:31:45.965631 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eacc667f7ce447f964655e8a7e8ce40 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:31:45.965675 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eacc667f7ce447f964655e8a7e8ce40 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b75bc97a2084e62a6c1d41b92ada830" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b75bc97a2084e62a6c1d41b92ada830" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } } peers { permanent_uuid: "8eacc667f7ce447f964655e8a7e8ce40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 } } peers { permanent_uuid: "ec5da7d4a27a4cc1a986b8d3bd4138aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } } }
I20260926 09:31:45.965734 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eacc667f7ce447f964655e8a7e8ce40 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:31:45.966017 23286 mvcc.cc:205] Tried to move back new op lower bound from 7333540273992830976 to 7333540273769504768. Current Snapshot: MvccSnapshot[applied={T|T < 7333540273809047552}]
I20260926 09:31:45.970539 23285 catalog_manager.cc:1259] Loaded cluster ID: 4b3c9ee8a76e4ad3b5e693a4cc1f7e9d
I20260926 09:31:45.970595 23285 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8eacc667f7ce447f964655e8a7e8ce40: loading cluster ID for follower catalog manager: success
I20260926 09:31:45.972939 23285 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8eacc667f7ce447f964655e8a7e8ce40: acquiring CA information for follower catalog manager: success
I20260926 09:31:45.973278 23285 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8eacc667f7ce447f964655e8a7e8ce40: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 09:31:45.979624 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:45.977978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41940 (local address 127.0.98.61:36207)
0926 09:31:45.978023 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:45.978025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:45.978929 (+ 904us) server_negotiation.cc:407] Connection header received
0926 09:31:45.979037 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:45.979038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:45.979050 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:45.979084 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:45.979547 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41940: BlockingRecv error: recv got EOF from 127.0.0.1:41940 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 09:31:45.980599 23228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:45.978325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35094 (local address 127.0.98.60:42287)
0926 09:31:45.978367 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:45.978369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:45.979829 (+ 1460us) server_negotiation.cc:407] Connection header received
0926 09:31:45.980111 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:45.980112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:45.980123 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:45.980153 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:45.980532 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35094: BlockingRecv error: recv got EOF from 127.0.0.1:35094 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 09:31:45.982025 23078 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:45.978677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55702 (local address 127.0.98.62:38155)
0926 09:31:45.978724 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:45.978726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:45.980010 (+ 1284us) server_negotiation.cc:407] Connection header received
0926 09:31:45.980076 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:45.980077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:45.980088 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:45.980143 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:45.981959 (+ 1816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55702: BlockingRecv error: recv got EOF from 127.0.0.1:55702 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:31:45.989502 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.985921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41944 (local address 127.0.98.61:36207)
0926 09:31:45.985960 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:45.985962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:45.986165 (+ 203us) server_negotiation.cc:407] Connection header received
0926 09:31:45.986271 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:45.986272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:45.986326 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:45.986371 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:45.986581 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.987378 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.988555 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.988852 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.988891 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.989112 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:45.989236 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:45.989295 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:45.989408 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 09:31:45.989428 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 09:31:45.992695 23228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:45.991887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35100 (local address 127.0.98.60:42287)
0926 09:31:45.991928 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:45.991930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:45.992129 (+ 199us) server_negotiation.cc:407] Connection header received
0926 09:31:45.992234 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:45.992235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:45.992253 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:45.992295 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:45.992639 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35100: BlockingRecv error: recv got EOF from 127.0.0.1:35100 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:31:45.998219 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:45.994879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41948 (local address 127.0.98.61:36207)
0926 09:31:45.994914 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:45.994916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:45.995161 (+ 245us) server_negotiation.cc:407] Connection header received
0926 09:31:45.995254 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:45.995255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:45.995263 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:45.995303 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:45.995471 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.996224 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.997351 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:45.997736 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:45.997777 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:45.997969 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:45.998001 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:45.998027 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:45.998126 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 09:31:45.998146 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 09:31:46.001299 23228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:46.000487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35106 (local address 127.0.98.60:42287)
0926 09:31:46.000524 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:46.000526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:46.000744 (+ 218us) server_negotiation.cc:407] Connection header received
0926 09:31:46.000840 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:46.000842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:46.000852 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:46.000882 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:46.001231 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35106: BlockingRecv error: recv got EOF from 127.0.0.1:35106 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:31:46.007558 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:46.004297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41960 (local address 127.0.98.61:36207)
0926 09:31:46.004343 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:46.004345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:46.004580 (+ 235us) server_negotiation.cc:407] Connection header received
0926 09:31:46.004673 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:46.004674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:46.004682 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:46.004709 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:46.004869 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:46.005662 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:46.006753 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:46.007042 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:46.007081 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:46.007275 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:46.007330 (+ 55us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:46.007358 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:46.007464 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 09:31:46.007481 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 09:31:46.010589 23228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:46.009828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35112 (local address 127.0.98.60:42287)
0926 09:31:46.009868 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:46.009870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:46.010067 (+ 197us) server_negotiation.cc:407] Connection header received
0926 09:31:46.010161 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:46.010162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:46.010173 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:46.010209 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:46.010536 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35112: BlockingRecv error: recv got EOF from 127.0.0.1:35112 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:31:46.019053 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:46.015701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41962 (local address 127.0.98.61:36207)
0926 09:31:46.015742 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:46.015743 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:46.016000 (+ 257us) server_negotiation.cc:407] Connection header received
0926 09:31:46.016097 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:46.016098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:46.016108 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:46.016137 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:46.016339 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:46.017085 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:46.018235 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:46.018549 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:46.018590 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:46.018787 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:46.018820 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:46.018848 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:46.018951 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 09:31:46.018969 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 09:31:46.022258 23228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:46.021432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35128 (local address 127.0.98.60:42287)
0926 09:31:46.021473 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:46.021475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:46.021721 (+ 246us) server_negotiation.cc:407] Connection header received
0926 09:31:46.021821 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:46.021822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:46.021833 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:46.021862 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:46.022197 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35128: BlockingRecv error: recv got EOF from 127.0.0.1:35128 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:31:46.035164 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:46.031700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41964 (local address 127.0.98.61:36207)
0926 09:31:46.031745 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:46.031747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:46.032059 (+ 312us) server_negotiation.cc:407] Connection header received
0926 09:31:46.032165 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:46.032166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:46.032177 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:46.032207 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:46.032404 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:46.033146 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:46.034371 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:46.034666 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:46.034704 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:46.034898 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:46.034931 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:46.034959 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:46.035063 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 09:31:46.035081 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 09:31:46.038451 23228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:46.037554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35134 (local address 127.0.98.60:42287)
0926 09:31:46.037626 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:46.037628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:46.037841 (+ 213us) server_negotiation.cc:407] Connection header received
0926 09:31:46.037933 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:46.037934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:46.037946 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:46.037975 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:46.038390 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35134: BlockingRecv error: recv got EOF from 127.0.0.1:35134 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:31:46.059048 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:46.055687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41974 (local address 127.0.98.61:36207)
0926 09:31:46.055731 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:46.055734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:46.056031 (+ 297us) server_negotiation.cc:407] Connection header received
0926 09:31:46.056132 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:46.056133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:46.056143 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:46.056173 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:46.056375 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:46.057070 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:46.058219 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:46.058501 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:46.058548 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:46.058762 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:46.058798 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:46.058829 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:46.058943 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 09:31:46.058965 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 09:31:46.062408 23228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:46.061504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35136 (local address 127.0.98.60:42287)
0926 09:31:46.061546 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:46.061548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:46.061811 (+ 263us) server_negotiation.cc:407] Connection header received
0926 09:31:46.061913 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:46.061914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:46.061926 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:46.061956 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:46.062345 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35136: BlockingRecv error: recv got EOF from 127.0.0.1:35136 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:31:46.098994 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:46.095797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.0.98.61:36207)
0926 09:31:46.095843 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:46.095846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:46.096019 (+ 173us) server_negotiation.cc:407] Connection header received
0926 09:31:46.096078 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:46.096079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:46.096090 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:46.096155 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:46.096312 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:46.097058 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:46.098160 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:46.098464 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:46.098538 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:46.098730 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:46.098765 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:46.098818 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:46.098889 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 09:31:46.098908 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 09:31:46.102293 23228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:46.100863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35148 (local address 127.0.98.60:42287)
0926 09:31:46.100907 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:46.100909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:46.101784 (+ 875us) server_negotiation.cc:407] Connection header received
0926 09:31:46.101860 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:46.101861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:46.101874 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:46.101939 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:46.102236 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35148: BlockingRecv error: recv got EOF from 127.0.0.1:35148 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:31:46.169987 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:46.167642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41998 (local address 127.0.98.61:36207)
0926 09:31:46.167719 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:46.167721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:46.167814 (+ 93us) server_negotiation.cc:407] Connection header received
0926 09:31:46.167847 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:46.167847 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:46.167872 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:46.167937 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:46.168105 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:46.168562 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:46.169326 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:46.169498 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:46.169546 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:46.169774 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:46.169800 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:46.169841 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:46.169909 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 09:31:46.169926 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 09:31:46.172497 23228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:46.171842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35152 (local address 127.0.98.60:42287)
0926 09:31:46.171905 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:46.171907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:46.172030 (+ 123us) server_negotiation.cc:407] Connection header received
0926 09:31:46.172074 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:46.172075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:46.172084 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:46.172132 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:46.172433 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35152: BlockingRecv error: recv got EOF from 127.0.0.1:35152 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 09:31:46.303799 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:46.301628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42002 (local address 127.0.98.61:36207)
0926 09:31:46.301707 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:46.301709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:46.301724 (+ 15us) server_negotiation.cc:407] Connection header received
0926 09:31:46.301825 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:46.301826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:46.301836 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:46.301889 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:46.302015 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:46.302483 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:46.303205 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:46.303395 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:46.303446 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:46.303564 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:46.303589 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:46.303627 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:46.303677 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 09:31:46.303692 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 09:31:46.306116 23228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:46.305474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35168 (local address 127.0.98.60:42287)
0926 09:31:46.305540 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:46.305542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:46.305625 (+ 83us) server_negotiation.cc:407] Connection header received
0926 09:31:46.305692 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:46.305693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:46.305703 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:46.305767 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:46.306058 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35168: BlockingRecv error: recv got EOF from 127.0.0.1:35168 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:31:46.512732 23244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b75bc97a2084e62a6c1d41b92ada830" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b75bc97a2084e62a6c1d41b92ada830" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } } peers { permanent_uuid: "8eacc667f7ce447f964655e8a7e8ce40" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36207 } } peers { permanent_uuid: "ec5da7d4a27a4cc1a986b8d3bd4138aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42287 } } }
I20260926 09:31:46.512902 23244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b75bc97a2084e62a6c1d41b92ada830 [sys.catalog]: This master's current role is: LEADER
I20260926 09:31:46.565641 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:46.563430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42010 (local address 127.0.98.61:36207)
0926 09:31:46.563501 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:46.563503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:46.563515 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:31:46.563615 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:46.563616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:46.563625 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:46.563692 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:46.563834 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:46.564304 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:46.565077 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:46.565252 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:46.565300 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:46.565434 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:46.565460 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:46.565497 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:46.565549 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 09:31:46.565564 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 09:31:46.567965 23228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:31:46.567364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35184 (local address 127.0.98.60:42287)
0926 09:31:46.567430 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:46.567432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:46.567443 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:31:46.567473 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:46.567474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:46.567484 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:46.567534 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:31:46.567883 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35184: BlockingRecv error: recv got EOF from 127.0.0.1:35184 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 09:31:46.890791 23241 catalog_manager.cc:1259] Loaded cluster ID: 4b3c9ee8a76e4ad3b5e693a4cc1f7e9d
I20260926 09:31:46.890893 23241 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ec5da7d4a27a4cc1a986b8d3bd4138aa: loading cluster ID for follower catalog manager: success
I20260926 09:31:46.891716 23241 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ec5da7d4a27a4cc1a986b8d3bd4138aa: acquiring CA information for follower catalog manager: success
I20260926 09:31:46.892028 23241 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ec5da7d4a27a4cc1a986b8d3bd4138aa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 09:31:47.083418 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:47.081075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42018 (local address 127.0.98.61:36207)
0926 09:31:47.081187 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:47.081189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:47.081259 (+ 70us) server_negotiation.cc:407] Connection header received
0926 09:31:47.081309 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:47.081310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:47.081321 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:47.081380 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:47.081492 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:47.082034 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:47.082777 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:47.082990 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:47.083055 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:47.083196 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:47.083227 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:47.083287 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:47.083324 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 09:31:47.083346 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":68,"threads_started":1}
I20260926 09:31:47.087697 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:47.085199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35192 (local address 127.0.98.60:42287)
0926 09:31:47.085301 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:47.085303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:47.085312 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:31:47.085362 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:47.085362 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:47.085436 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:47.085520 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:47.085653 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:47.086142 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:47.086867 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:47.087090 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:47.087166 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:47.087340 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:47.087459 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:47.087557 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:47.087575 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 09:31:47.087599 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":54,"threads_started":1}
I20260926 09:31:47.092701 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:47.089530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42034 (local address 127.0.98.61:36207)
0926 09:31:47.089610 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:47.089612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:47.089886 (+ 274us) server_negotiation.cc:407] Connection header received
0926 09:31:47.089954 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:47.089955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:47.089970 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:47.090052 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:47.090204 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:47.090779 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:47.091754 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:47.092107 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:47.092182 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:47.092382 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:47.092421 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:47.092492 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:47.092563 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 09:31:47.092591 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 09:31:47.093134 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:47.089809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35202 (local address 127.0.98.60:42287)
0926 09:31:47.089854 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:47.089856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:47.089963 (+ 107us) server_negotiation.cc:407] Connection header received
0926 09:31:47.090062 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:47.090063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:47.090074 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:47.090129 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:47.090302 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:47.091126 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:47.092295 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:47.092605 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:47.092667 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:47.092832 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:47.092864 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:47.092893 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:47.093010 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 09:31:47.093058 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 09:31:47.093146 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:31:47.089533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55714 (local address 127.0.98.62:38155)
0926 09:31:47.089683 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 09:31:47.089686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:31:47.089765 (+ 79us) server_negotiation.cc:407] Connection header received
0926 09:31:47.089876 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:31:47.089877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:31:47.089967 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:31:47.090032 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:31:47.090421 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:47.091301 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:47.092146 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:31:47.092447 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:31:47.092509 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:31:47.092767 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:31:47.092887 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:31:47.092927 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 09:31:47.093044 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 09:31:47.093071 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260926 09:31:47.099124 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 23006
I20260926 09:31:47.104560 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 23081
I20260926 09:31:47.109338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNX8gJ6/build/release/bin/kudu with pid 23156
2026-09-26T09:31:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2838 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5705 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (268046 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST